Jul 22 04:31:02.363341 ip-10-0-142-47 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 04:31:02.363350 ip-10-0-142-47 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 04:31:02.363357 ip-10-0-142-47 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 04:31:02.363567 ip-10-0-142-47 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 04:31:13.700161 ip-10-0-142-47 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 04:31:13.700183 ip-10-0-142-47 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d8ae6e00228545a186f107f3e958ae0f -- Jul 22 04:33:08.166744 ip-10-0-142-47 systemd[1]: Starting Kubernetes Kubelet... Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.556927 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561715 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561738 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561744 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561748 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561751 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561756 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561759 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 04:33:08.656395 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561762 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 04:33:08.631801 ip-10-0-142-47 systemd[1]: Started Kubernetes Kubelet. Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561766 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561770 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561773 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561776 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561780 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561786 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561790 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561793 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561798 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561800 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561803 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561806 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561810 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561813 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561815 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561821 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561823 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561826 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 04:33:08.657495 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561831 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561834 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561837 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561840 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561844 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561847 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561850 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561854 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561857 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561860 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561865 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561867 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561878 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561881 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561884 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561889 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561892 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561894 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561899 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 04:33:08.658527 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561902 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561906 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561912 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561915 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561918 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561923 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561925 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561928 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561933 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561937 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561939 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561944 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561947 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561950 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561954 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561957 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561960 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561966 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561970 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 04:33:08.659483 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561973 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561976 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561979 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561982 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561986 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561989 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561992 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.561994 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562000 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562003 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562005 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562010 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562012 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562015 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562019 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562022 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562025 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562027 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562031 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562034 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 04:33:08.660175 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562038 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562042 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562045 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562048 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562051 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562056 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562059 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562061 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562066 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562068 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562071 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562073 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562079 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562082 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.562089 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564790 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564804 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564808 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564811 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 04:33:08.660947 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564814 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564816 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564819 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564821 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564825 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564828 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564830 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564833 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564836 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564839 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564841 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564844 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564848 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564850 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564853 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564855 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564858 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564861 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564864 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564866 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 04:33:08.661733 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564868 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564871 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564873 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564876 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564881 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564884 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564888 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564891 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564894 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564897 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564900 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564902 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564905 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564907 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564910 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564913 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564915 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564918 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564920 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564923 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 04:33:08.662459 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564926 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564928 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564932 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564935 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564938 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564941 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564944 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564947 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564949 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564952 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564954 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564957 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564959 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564962 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564965 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564967 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564970 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564972 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564975 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564979 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 04:33:08.663130 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564982 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564986 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564989 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564991 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564994 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564997 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.564999 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565002 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565005 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565007 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565010 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565012 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565016 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565018 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565021 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565024 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565026 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565029 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565031 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 04:33:08.663875 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565034 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565037 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565039 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565042 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565044 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565047 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565051 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565055 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565059 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565062 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565065 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565068 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565071 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565074 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565076 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565082 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565508 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565516 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565520 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 04:33:08.665299 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565524 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565527 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565530 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565533 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565536 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565538 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565541 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565543 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565546 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565549 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565551 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565554 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565557 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565560 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565563 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565566 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565569 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565571 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565573 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565576 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 04:33:08.665851 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565579 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565581 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565583 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565586 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565591 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565593 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565596 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565598 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565602 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565605 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565607 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565610 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565613 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565615 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565618 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565620 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565624 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565628 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565630 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 04:33:08.890913 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565633 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565636 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565639 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565642 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565645 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565648 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565650 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565654 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565656 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565659 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565662 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565664 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565667 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565669 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565672 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565675 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565678 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565681 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565685 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565687 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 04:33:08.916739 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565690 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565694 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565696 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565699 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565701 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565704 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565707 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565709 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565712 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565716 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565719 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565723 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565726 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565728 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565731 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565733 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565736 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565739 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565741 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 04:33:08.917544 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565743 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565747 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565749 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565752 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565755 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565757 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565760 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565762 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565765 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565768 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565772 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565774 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565776 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565779 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565783 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565785 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565788 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.565790 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565869 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565877 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565884 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 04:33:08.918205 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565888 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565893 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565896 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565900 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565905 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565908 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565911 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565914 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565917 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565926 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565929 2572 flags.go:64] FLAG: --cgroup-root="" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565932 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565935 2572 flags.go:64] FLAG: --client-ca-file="" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565938 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565942 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565946 2572 flags.go:64] FLAG: --cluster-domain="" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565949 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565952 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565955 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565959 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565968 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565973 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565976 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 04:33:08.919032 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565980 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565983 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565987 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565990 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565993 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.565996 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566001 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566004 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566007 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566010 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566012 2572 flags.go:64] FLAG: --enable-server="true" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566015 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566027 2572 flags.go:64] FLAG: --event-burst="100" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566030 2572 flags.go:64] FLAG: --event-qps="50" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566033 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566036 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566039 2572 flags.go:64] FLAG: --eviction-hard="" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566048 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566051 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566054 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566057 2572 flags.go:64] FLAG: --eviction-soft="" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566060 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566063 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566067 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566070 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 04:33:08.919751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566072 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566075 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566078 2572 flags.go:64] FLAG: --feature-gates="" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566082 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566085 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566090 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566093 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566096 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566099 2572 flags.go:64] FLAG: --help="false" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566103 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-142-47.ec2.internal" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566107 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566110 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566113 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566116 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566119 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566122 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566125 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566128 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566131 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566134 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566137 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566140 2572 flags.go:64] FLAG: --kube-reserved="" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566143 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566146 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 04:33:08.920742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566149 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566151 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566154 2572 flags.go:64] FLAG: --lock-file="" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566157 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566160 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566163 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566169 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566172 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566175 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566178 2572 flags.go:64] FLAG: --logging-format="text" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566181 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566184 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566187 2572 flags.go:64] FLAG: --manifest-url="" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566192 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566197 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566200 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566204 2572 flags.go:64] FLAG: --max-pods="110" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566208 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566211 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566215 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566218 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566221 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566224 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566227 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566235 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 04:33:08.921473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566238 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566241 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566244 2572 flags.go:64] FLAG: --pod-cidr="" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566247 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566252 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566255 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566258 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566261 2572 flags.go:64] FLAG: --port="10250" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566264 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566266 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02de99e207029a4ba" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566270 2572 flags.go:64] FLAG: --qos-reserved="" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566272 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566276 2572 flags.go:64] FLAG: --register-node="true" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566279 2572 flags.go:64] FLAG: --register-with-taints="" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566283 2572 flags.go:64] FLAG: --registry-burst="10" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566286 2572 flags.go:64] FLAG: --registry-qps="5" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566289 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566292 2572 flags.go:64] FLAG: --reserved-memory="" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566295 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566299 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566303 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566306 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566308 2572 flags.go:64] FLAG: --runonce="false" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566311 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566314 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 04:33:08.922394 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566319 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566322 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566325 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566328 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566331 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566334 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566337 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566340 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566343 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566346 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566349 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566352 2572 flags.go:64] FLAG: --system-cgroups="" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566355 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566361 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566363 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566366 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566371 2572 flags.go:64] FLAG: --tls-min-version="" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566387 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566390 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566393 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566395 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566399 2572 flags.go:64] FLAG: --v="2" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566403 2572 flags.go:64] FLAG: --version="false" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566407 2572 flags.go:64] FLAG: --vmodule="" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566412 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 04:33:08.923256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566415 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566551 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566556 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566560 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566563 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566566 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566568 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566572 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566575 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566578 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566580 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566583 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566586 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566589 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566591 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566594 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566598 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566601 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566604 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566607 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 04:33:08.926126 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566610 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566613 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566615 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566618 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566621 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566624 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566627 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566630 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566633 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566636 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566638 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566642 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566645 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566647 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566651 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566654 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566656 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566659 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566662 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566666 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 04:33:08.926813 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566669 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566672 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566674 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566677 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566679 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566682 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566686 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566689 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566693 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566696 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566698 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566701 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566704 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566708 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566711 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566713 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566716 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566719 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566721 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 04:33:08.927718 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566724 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566726 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566729 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566732 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566734 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566738 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566741 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566745 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566748 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566750 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566753 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566756 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566759 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566762 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566765 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566768 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566770 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566773 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566776 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566778 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 04:33:08.928295 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566781 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566784 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566786 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566789 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566791 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566794 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566797 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566800 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566802 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566804 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566807 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566809 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566812 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566815 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566817 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566820 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566823 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566826 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566829 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566833 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 04:33:08.929073 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.566836 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.566841 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.573453 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.573470 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.574099 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.574107 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574166 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574171 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574175 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574179 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574182 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574185 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574188 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574191 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574194 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 04:33:08.929697 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574196 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574199 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574202 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574204 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574209 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574212 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574216 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574218 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574221 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574224 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574226 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574229 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574232 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574234 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574237 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574245 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574248 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574251 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574253 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 04:33:09.188313 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574256 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574259 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574261 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574264 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574267 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574271 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574274 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574278 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574281 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574284 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574287 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574290 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574293 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574296 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574299 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574302 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574305 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574308 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574311 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574314 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 04:33:09.189403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574317 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574319 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574322 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574325 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574331 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574334 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574337 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574339 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574342 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574346 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574350 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574353 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574355 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574358 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574361 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574364 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574367 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574369 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574384 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 04:33:09.190273 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574387 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574408 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574416 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574419 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574421 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574425 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574427 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574430 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574433 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574436 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574439 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574442 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574444 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574447 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574450 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574452 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574455 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574457 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574460 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574463 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 04:33:09.190864 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574466 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574468 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574471 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574475 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574477 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574481 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574484 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574486 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574489 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574491 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574494 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574497 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.574502 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574624 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574630 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 04:33:09.191780 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574633 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574636 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574639 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574642 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574644 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574648 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574650 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574654 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574656 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574659 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574662 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574664 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574669 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574672 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574675 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574679 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574682 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574686 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574689 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574691 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 04:33:09.192279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574694 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574698 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574701 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574706 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574709 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574712 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574715 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574718 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574721 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574724 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574727 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574730 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574732 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574735 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574738 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574740 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574743 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574746 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574748 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574751 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 04:33:09.193069 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574753 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574756 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574759 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574761 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574765 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574768 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574770 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574773 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574775 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574778 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574780 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574783 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574785 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574788 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574791 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574795 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574798 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574800 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574803 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 04:33:09.193691 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574805 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574807 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574810 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574813 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574815 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574818 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574820 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574823 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574826 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574828 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574830 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574833 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574835 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574838 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574841 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574844 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574846 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574850 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574852 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 04:33:09.194466 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574855 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574859 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574861 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574864 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574867 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574869 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574872 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574874 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574876 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574881 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574884 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574886 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574889 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574891 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574894 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574896 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574899 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574901 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 04:33:09.195051 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:08.574904 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.574909 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.575012 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.577688 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.578971 2572 server.go:1019] "Starting client certificate rotation" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.579072 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.579113 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.581101 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.582226 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.591720 2572 log.go:25] "Validated CRI v1 runtime API" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.596413 2572 log.go:25] "Validated CRI v1 image API" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.597236 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 04:33:09.195867 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.602992 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a6f0081-f6f7-4b21-8690-6cdb9cbb7649:/dev/nvme0n1p4 eca399d6-02a3-404e-967c-5b7777d0cce9:/dev/nvme0n1p3] Jul 22 04:33:09.196269 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.603009 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 04:33:09.196269 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.608892 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.610715 2572 manager.go:216] Machine: {Timestamp:2026-07-22 04:33:08.60788273 +0000 UTC m=+0.326487858 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091353 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ef1df3724f4b9633a4dc9ef1e1314 SystemUUID:ec2ef1df-3724-f4b9-633a-4dc9ef1e1314 BootID:d8ae6e00-2285-45a1-86f1-07f3e958ae0f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:72:12:31:5f:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:72:12:31:5f:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:12:90:b7:ee:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.610833 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.610915 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.613424 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.613447 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-47.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.615421 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.615432 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.615439 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.615979 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.616944 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620332 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620454 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620480 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620488 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620503 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620513 2572 kubelet.go:398] "Adding apiserver pod source" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620524 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 04:33:09.417110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620867 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.620885 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.622885 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.624682 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.624720 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.625257 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.625272 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.625282 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.625291 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.625301 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.625309 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.626779 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.626788 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.626796 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.626805 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.626834 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.626846 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.627486 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.627500 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.627501 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kppzz" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.630593 2572 server.go:1268] "Started kubelet" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.630695 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.631009 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.631117 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.632294 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.632651 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-47.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.632693 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.632716 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.635693 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kppzz" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.635963 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.637970 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.637244 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-47.ec2.internal.18c482745cd0da5c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-47.ec2.internal,UID:ip-10-0-142-47.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-47.ec2.internal,},FirstTimestamp:2026-07-22 04:33:08.630567516 +0000 UTC m=+0.349172643,LastTimestamp:2026-07-22 04:33:08.630567516 +0000 UTC m=+0.349172643,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-47.ec2.internal,}" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.645003 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.645019 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.645084 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.645641 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.645657 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.645752 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.645784 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.645792 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 22 04:33:09.417785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.646032 2572 factory.go:55] Registering systemd factory Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.646045 2572 factory.go:223] Registration of the systemd container factory successfully Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.646222 2572 factory.go:153] Registering CRI-O factory Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.646233 2572 factory.go:223] Registration of the crio container factory successfully Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.646274 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.646295 2572 factory.go:103] Registering Raw factory Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.646310 2572 manager.go:1196] Started watching for new ooms in manager Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.646705 2572 manager.go:349] Starting recovery of all containers Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.650036 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.652937 2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-47.ec2.internal\" not found" node="ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.659832 2572 manager.go:354] Recovery completed Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.664170 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.665949 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientMemory" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.665976 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.665987 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientPID" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.666583 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.666592 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.666592 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.666660 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.667299 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientMemory" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.667323 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.667332 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientPID" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.669202 2572 policy_none.go:49] "None policy: Start" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.669215 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.669224 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.671142 2572 policy_none.go:47] "Start" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.710935 2572 manager.go:337] "Starting Device Plugin manager" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.711108 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.711120 2572 server.go:87] "Starting device plugin registration server" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.711326 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.711354 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.711445 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.711518 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.711526 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.712099 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.712151 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.793136 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.794350 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.794392 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.794415 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.794447 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.797049 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.811972 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.812771 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientMemory" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.812796 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.812806 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientPID" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.812828 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.820352 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.820370 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-47.ec2.internal\": node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.419277 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.835112 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.897811 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal"] Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.897867 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.899205 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientMemory" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.899235 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.899245 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientPID" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.902154 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.902245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.902272 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.902899 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientMemory" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.902912 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientMemory" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.902929 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.902944 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientPID" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.902933 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.903085 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientPID" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.906127 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.906153 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.906832 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientMemory" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.906855 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.906865 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeHasSufficientPID" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.936060 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.936237 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-47.ec2.internal\" not found" node="ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:08.940924 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-47.ec2.internal\" not found" node="ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.947186 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/480d41ccf0f1f19198559f5c186d8771-config\") pod \"kube-apiserver-proxy-ip-10-0-142-47.ec2.internal\" (UID: \"480d41ccf0f1f19198559f5c186d8771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.947209 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dec19784c32143c70c3ceb1f215fa236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal\" (UID: \"dec19784c32143c70c3ceb1f215fa236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:08.947230 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dec19784c32143c70c3ceb1f215fa236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal\" (UID: \"dec19784c32143c70c3ceb1f215fa236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.036598 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.047735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/480d41ccf0f1f19198559f5c186d8771-config\") pod \"kube-apiserver-proxy-ip-10-0-142-47.ec2.internal\" (UID: \"480d41ccf0f1f19198559f5c186d8771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.047764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dec19784c32143c70c3ceb1f215fa236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal\" (UID: \"dec19784c32143c70c3ceb1f215fa236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.420697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.047781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dec19784c32143c70c3ceb1f215fa236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal\" (UID: \"dec19784c32143c70c3ceb1f215fa236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.421620 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.047822 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dec19784c32143c70c3ceb1f215fa236-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal\" (UID: \"dec19784c32143c70c3ceb1f215fa236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.421620 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.047840 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/480d41ccf0f1f19198559f5c186d8771-config\") pod \"kube-apiserver-proxy-ip-10-0-142-47.ec2.internal\" (UID: \"480d41ccf0f1f19198559f5c186d8771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.421620 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.047855 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dec19784c32143c70c3ceb1f215fa236-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal\" (UID: \"dec19784c32143c70c3ceb1f215fa236\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.421620 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.136942 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.421620 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.238575 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.421620 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.238622 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.421620 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.245141 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.421620 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.342854 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.443459 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-47.ec2.internal\" not found" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.456557 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.505543 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.545221 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.555746 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.556813 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.566795 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.578808 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.578955 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.578955 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.578963 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.578963 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.623456 2572 apiserver.go:52] "Watching apiserver" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.641143 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 04:28:09 +0000 UTC" deadline="2028-04-07 10:46:57.656712032 +0000 UTC" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.641170 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15006h13m48.015545895s" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.643155 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.643559 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-k8rb5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal","openshift-multus/network-metrics-daemon-wsxd4","openshift-network-operator/iptables-alerter-drhg7","kube-system/global-pull-secret-syncer-hrkws","openshift-image-registry/node-ca-bmptl","openshift-multus/multus-additional-cni-plugins-9nnqb","openshift-multus/multus-zxf5g","openshift-network-diagnostics/network-check-target-scr4t","openshift-ovn-kubernetes/ovnkube-node-bpv66","kube-system/konnectivity-agent-t5hss","kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r","openshift-cluster-node-tuning-operator/tuned-rdz55"] Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.645105 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.646182 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.648811 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.648895 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.648985 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.649480 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.649712 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.649715 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-75jxj\"" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.649813 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kltj\" (UniqueName: \"kubernetes.io/projected/a1fff17b-28b4-43d6-b045-842423eceb38-kube-api-access-5kltj\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.671094 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651634 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651648 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9vq\" (UniqueName: \"kubernetes.io/projected/72bb9cb1-2e8a-44c3-8d61-043b7605a224-kube-api-access-tz9vq\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651705 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-device-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651748 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cnibin\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651807 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651835 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651896 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-registration-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651941 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-sys-fs\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651965 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-system-cni-dir\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.652011 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-os-release\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.652066 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.651492 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.652104 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.654855 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.654869 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-socket-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.654908 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.672522 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.655202 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.655288 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.655396 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cbz8r\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.657465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.659725 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.659731 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dj76d\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.659768 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.659817 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.659878 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.660025 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.660025 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.662247 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.664404 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.664504 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.664550 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.664581 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.664652 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w92rg\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.664579 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.666874 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.669234 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.669606 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.669821 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q4mvr\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.669960 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.671571 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-czspn\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.671825 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.672001 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.672161 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.674091 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.674993 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.675055 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.675155 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p287c\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.675485 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.675541 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 04:33:09.861910 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.675550 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.675554 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.676521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.676642 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.676651 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f99sg\"" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.676685 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.678606 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.678934 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4kz8c\"" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.684967 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dvw2q" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.691186 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dvw2q" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.746580 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.755828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.755861 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-cni-netd\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.755886 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66acea06-2cd5-4939-bca5-b63eda7f5b29-agent-certs\") pod \"konnectivity-agent-t5hss\" (UID: \"66acea06-2cd5-4939-bca5-b63eda7f5b29\") " pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.755910 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysconfig\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.755944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-hostroot\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.755966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-var-lib-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.755996 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-etc-selinux\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.755998 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-ovn\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756037 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-ovnkube-config\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb19ef52-a0fa-4b32-8b32-693739754b16-iptables-alerter-script\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756075 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c051ba5-d2d1-4add-89e0-591295ecaf5e-dbus\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.863412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756094 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cnibin\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756171 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-run-netns\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756195 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cnibin\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756196 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-cni-bin\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756246 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-ovnkube-script-lib\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756271 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-kubernetes\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756296 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-k8s-cni-cncf-io\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756322 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-log-socket\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756374 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-kubelet\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756431 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb19ef52-a0fa-4b32-8b32-693739754b16-host-slash\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-socket-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756483 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d00b0010-35ef-4ad0-a483-937a188385cc-host\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756505 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-node-log\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756529 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756551 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-run\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.864513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-sys\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756593 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-socket-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-tuned\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756645 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-os-release\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756667 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-cnibin\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756698 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-socket-dir-parent\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-registration-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756786 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-registration-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756813 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-cni-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-os-release\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756863 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-systemd\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wxz6\" (UniqueName: \"kubernetes.io/projected/6855994a-275d-47be-b3c7-0071685781c0-kube-api-access-8wxz6\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756916 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-os-release\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756923 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66acea06-2cd5-4939-bca5-b63eda7f5b29-konnectivity-ca\") pod \"konnectivity-agent-t5hss\" (UID: \"66acea06-2cd5-4939-bca5-b63eda7f5b29\") " pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756950 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysctl-d\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756965 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63218f68-800c-4699-930f-b228bb95d1f4-cni-binary-copy\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.865697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756980 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-systemd-units\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.756994 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-modprobe-d\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757008 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysctl-conf\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757023 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-cni-bin\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757081 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9vq\" (UniqueName: \"kubernetes.io/projected/72bb9cb1-2e8a-44c3-8d61-043b7605a224-kube-api-access-tz9vq\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757104 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-device-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/227413da-239d-4d90-8c76-8861f42e0ac0-hosts-file\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757152 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-cni-multus\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757174 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63218f68-800c-4699-930f-b228bb95d1f4-multus-daemon-config\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757229 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-device-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757238 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-sys-fs\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227413da-239d-4d90-8c76-8861f42e0ac0-tmp-dir\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757292 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757301 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1fff17b-28b4-43d6-b045-842423eceb38-sys-fs\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757311 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-system-cni-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.866662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757368 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-conf-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757409 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbmx\" (UniqueName: \"kubernetes.io/projected/0c2de1e1-0496-4eda-b5e5-12b1c181582e-kube-api-access-6gbmx\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-system-cni-dir\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757469 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-etc-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757495 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-var-lib-kubelet\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757501 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-system-cni-dir\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757520 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmmc\" (UniqueName: \"kubernetes.io/projected/7bbc22f3-78b5-4c69-b089-ea405855ee65-kube-api-access-qlmmc\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c051ba5-d2d1-4add-89e0-591295ecaf5e-kubelet-config\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757620 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kltj\" (UniqueName: \"kubernetes.io/projected/a1fff17b-28b4-43d6-b045-842423eceb38-kube-api-access-5kltj\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757629 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d00b0010-35ef-4ad0-a483-937a188385cc-serviceca\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757665 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757687 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-lib-modules\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757709 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-host\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.867865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6855994a-275d-47be-b3c7-0071685781c0-ovn-node-metrics-cert\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757778 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/72bb9cb1-2e8a-44c3-8d61-043b7605a224-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-systemd\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-netns\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-multus-certs\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757890 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9cjs\" (UniqueName: \"kubernetes.io/projected/63218f68-800c-4699-930f-b228bb95d1f4-kube-api-access-s9cjs\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757917 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5hd\" (UniqueName: \"kubernetes.io/projected/cb19ef52-a0fa-4b32-8b32-693739754b16-kube-api-access-gs5hd\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.757972 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-slash\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.758018 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-env-overrides\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.758107 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfzt\" (UniqueName: \"kubernetes.io/projected/227413da-239d-4d90-8c76-8861f42e0ac0-kube-api-access-tlfzt\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.758139 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbc22f3-78b5-4c69-b089-ea405855ee65-tmp\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.758162 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-etc-kubernetes\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.758187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xnn\" (UniqueName: \"kubernetes.io/projected/d00b0010-35ef-4ad0-a483-937a188385cc-kube-api-access-66xnn\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.758210 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-kubelet\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.868960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.758213 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.758380 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/72bb9cb1-2e8a-44c3-8d61-043b7605a224-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.763588 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.766553 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kltj\" (UniqueName: \"kubernetes.io/projected/a1fff17b-28b4-43d6-b045-842423eceb38-kube-api-access-5kltj\") pod \"aws-ebs-csi-driver-node-zdf4r\" (UID: \"a1fff17b-28b4-43d6-b045-842423eceb38\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.766562 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9vq\" (UniqueName: \"kubernetes.io/projected/72bb9cb1-2e8a-44c3-8d61-043b7605a224-kube-api-access-tz9vq\") pod \"multus-additional-cni-plugins-9nnqb\" (UID: \"72bb9cb1-2e8a-44c3-8d61-043b7605a224\") " pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-cni-netd\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863760 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66acea06-2cd5-4939-bca5-b63eda7f5b29-agent-certs\") pod \"konnectivity-agent-t5hss\" (UID: \"66acea06-2cd5-4939-bca5-b63eda7f5b29\") " pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysconfig\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863793 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-hostroot\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-var-lib-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863850 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-ovn\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-ovnkube-config\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863869 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-hostroot\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863885 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-var-lib-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863889 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb19ef52-a0fa-4b32-8b32-693739754b16-iptables-alerter-script\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863938 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-cni-netd\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863936 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c051ba5-d2d1-4add-89e0-591295ecaf5e-dbus\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.870046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.863977 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864004 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-run-netns\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864031 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-cni-bin\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864039 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysconfig\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864058 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-ovnkube-script-lib\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864083 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-kubernetes\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864088 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-ovn\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864094 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9c051ba5-d2d1-4add-89e0-591295ecaf5e-dbus\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864140 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-kubernetes\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.864143 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864172 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-k8s-cni-cncf-io\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864177 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-cni-bin\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864143 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-run-netns\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864201 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864241 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-k8s-cni-cncf-io\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.864255 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret podName:9c051ba5-d2d1-4add-89e0-591295ecaf5e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:10.364211088 +0000 UTC m=+2.082816225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret") pod "global-pull-secret-syncer-hrkws" (UID: "9c051ba5-d2d1-4add-89e0-591295ecaf5e") : object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.864282 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864295 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-log-socket\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.871359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864343 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-log-socket\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.864998 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs podName:0c2de1e1-0496-4eda-b5e5-12b1c181582e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:10.364972131 +0000 UTC m=+2.083577262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs") pod "network-metrics-daemon-wsxd4" (UID: "0c2de1e1-0496-4eda-b5e5-12b1c181582e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864981 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cb19ef52-a0fa-4b32-8b32-693739754b16-iptables-alerter-script\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.864442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-kubelet\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865051 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-ovnkube-script-lib\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865078 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb19ef52-a0fa-4b32-8b32-693739754b16-host-slash\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865135 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb19ef52-a0fa-4b32-8b32-693739754b16-host-slash\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865139 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d00b0010-35ef-4ad0-a483-937a188385cc-host\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865149 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-kubelet\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865185 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-node-log\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865233 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-node-log\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865187 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d00b0010-35ef-4ad0-a483-937a188385cc-host\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865256 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-run\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865295 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865310 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-run\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865339 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-sys\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.872407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-tuned\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865455 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-os-release\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865498 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-cnibin\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865527 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-sys\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-socket-dir-parent\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865608 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-cni-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865613 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-socket-dir-parent\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-systemd\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865669 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-cni-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865685 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wxz6\" (UniqueName: \"kubernetes.io/projected/6855994a-275d-47be-b3c7-0071685781c0-kube-api-access-8wxz6\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865719 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66acea06-2cd5-4939-bca5-b63eda7f5b29-konnectivity-ca\") pod \"konnectivity-agent-t5hss\" (UID: \"66acea06-2cd5-4939-bca5-b63eda7f5b29\") " pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysctl-d\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63218f68-800c-4699-930f-b228bb95d1f4-cni-binary-copy\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-systemd-units\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865848 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-modprobe-d\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysctl-conf\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.865904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-cni-bin\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.873376 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866041 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysctl-d\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866443 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/66acea06-2cd5-4939-bca5-b63eda7f5b29-konnectivity-ca\") pod \"konnectivity-agent-t5hss\" (UID: \"66acea06-2cd5-4939-bca5-b63eda7f5b29\") " pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866518 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-cni-bin\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866606 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63218f68-800c-4699-930f-b228bb95d1f4-cni-binary-copy\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866681 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-systemd-units\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866783 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-modprobe-d\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866906 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-sysctl-conf\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866917 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-ovnkube-config\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.866959 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-systemd\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867022 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-cnibin\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867070 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-os-release\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/227413da-239d-4d90-8c76-8861f42e0ac0-hosts-file\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867154 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-cni-multus\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867190 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63218f68-800c-4699-930f-b228bb95d1f4-multus-daemon-config\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867226 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227413da-239d-4d90-8c76-8861f42e0ac0-tmp-dir\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-system-cni-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867287 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-conf-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867323 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbmx\" (UniqueName: \"kubernetes.io/projected/0c2de1e1-0496-4eda-b5e5-12b1c181582e-kube-api-access-6gbmx\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:09.874138 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-etc-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867402 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-var-lib-kubelet\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmmc\" (UniqueName: \"kubernetes.io/projected/7bbc22f3-78b5-4c69-b089-ea405855ee65-kube-api-access-qlmmc\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867495 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c051ba5-d2d1-4add-89e0-591295ecaf5e-kubelet-config\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d00b0010-35ef-4ad0-a483-937a188385cc-serviceca\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867574 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867605 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-lib-modules\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-host\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867669 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6855994a-275d-47be-b3c7-0071685781c0-ovn-node-metrics-cert\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867739 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-systemd\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867764 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63218f68-800c-4699-930f-b228bb95d1f4-multus-daemon-config\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867775 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-netns\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867809 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-multus-certs\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9cjs\" (UniqueName: \"kubernetes.io/projected/63218f68-800c-4699-930f-b228bb95d1f4-kube-api-access-s9cjs\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5hd\" (UniqueName: \"kubernetes.io/projected/cb19ef52-a0fa-4b32-8b32-693739754b16-kube-api-access-gs5hd\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867883 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-var-lib-cni-multus\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.874765 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-slash\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867948 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-run-ovn-kubernetes\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867950 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-env-overrides\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868085 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-lib-modules\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868155 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-host\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868209 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227413da-239d-4d90-8c76-8861f42e0ac0-tmp-dir\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868274 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-systemd\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-system-cni-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868387 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-multus-conf-dir\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-etc-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868763 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7bbc22f3-78b5-4c69-b089-ea405855ee65-var-lib-kubelet\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.867840 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/227413da-239d-4d90-8c76-8861f42e0ac0-hosts-file\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868935 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-netns\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868999 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-host-run-multus-certs\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869031 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9c051ba5-d2d1-4add-89e0-591295ecaf5e-kubelet-config\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869276 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-slash\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869488 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d00b0010-35ef-4ad0-a483-937a188385cc-serviceca\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.868211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-run-openvswitch\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.875358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869524 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfzt\" (UniqueName: \"kubernetes.io/projected/227413da-239d-4d90-8c76-8861f42e0ac0-kube-api-access-tlfzt\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869566 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbc22f3-78b5-4c69-b089-ea405855ee65-tmp\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-etc-kubernetes\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869627 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xnn\" (UniqueName: \"kubernetes.io/projected/d00b0010-35ef-4ad0-a483-937a188385cc-kube-api-access-66xnn\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-kubelet\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869733 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6855994a-275d-47be-b3c7-0071685781c0-host-kubelet\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.869907 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6855994a-275d-47be-b3c7-0071685781c0-env-overrides\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.870065 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63218f68-800c-4699-930f-b228bb95d1f4-etc-kubernetes\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.872753 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/66acea06-2cd5-4939-bca5-b63eda7f5b29-agent-certs\") pod \"konnectivity-agent-t5hss\" (UID: \"66acea06-2cd5-4939-bca5-b63eda7f5b29\") " pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.873382 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbc22f3-78b5-4c69-b089-ea405855ee65-tmp\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.873497 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6855994a-275d-47be-b3c7-0071685781c0-ovn-node-metrics-cert\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.876202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.873704 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7bbc22f3-78b5-4c69-b089-ea405855ee65-etc-tuned\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.876740 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.876683 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 04:33:09.876740 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.876705 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 04:33:09.876740 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.876718 2572 projected.go:196] Error preparing data for projected volume kube-api-access-747q7 for pod openshift-network-diagnostics/network-check-target-scr4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:09.876876 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:09.876784 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7 podName:eda46870-2792-446a-8583-862286c89ab9 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:10.376768169 +0000 UTC m=+2.095373303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-747q7" (UniqueName: "kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7") pod "network-check-target-scr4t" (UID: "eda46870-2792-446a-8583-862286c89ab9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:09.879556 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.879495 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5hd\" (UniqueName: \"kubernetes.io/projected/cb19ef52-a0fa-4b32-8b32-693739754b16-kube-api-access-gs5hd\") pod \"iptables-alerter-drhg7\" (UID: \"cb19ef52-a0fa-4b32-8b32-693739754b16\") " pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:09.881029 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.881000 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbmx\" (UniqueName: \"kubernetes.io/projected/0c2de1e1-0496-4eda-b5e5-12b1c181582e-kube-api-access-6gbmx\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:09.881137 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.881045 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9cjs\" (UniqueName: \"kubernetes.io/projected/63218f68-800c-4699-930f-b228bb95d1f4-kube-api-access-s9cjs\") pod \"multus-zxf5g\" (UID: \"63218f68-800c-4699-930f-b228bb95d1f4\") " pod="openshift-multus/multus-zxf5g" Jul 22 04:33:09.881481 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.881443 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfzt\" (UniqueName: \"kubernetes.io/projected/227413da-239d-4d90-8c76-8861f42e0ac0-kube-api-access-tlfzt\") pod \"node-resolver-k8rb5\" (UID: \"227413da-239d-4d90-8c76-8861f42e0ac0\") " pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:09.881568 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.881550 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wxz6\" (UniqueName: \"kubernetes.io/projected/6855994a-275d-47be-b3c7-0071685781c0-kube-api-access-8wxz6\") pod \"ovnkube-node-bpv66\" (UID: \"6855994a-275d-47be-b3c7-0071685781c0\") " pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:09.881621 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.881590 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xnn\" (UniqueName: \"kubernetes.io/projected/d00b0010-35ef-4ad0-a483-937a188385cc-kube-api-access-66xnn\") pod \"node-ca-bmptl\" (UID: \"d00b0010-35ef-4ad0-a483-937a188385cc\") " pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:09.882390 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.882370 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmmc\" (UniqueName: \"kubernetes.io/projected/7bbc22f3-78b5-4c69-b089-ea405855ee65-kube-api-access-qlmmc\") pod \"tuned-rdz55\" (UID: \"7bbc22f3-78b5-4c69-b089-ea405855ee65\") " pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:09.965279 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:09.965232 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec19784c32143c70c3ceb1f215fa236.slice/crio-97b0640e8ef12801914485ee6b21e6410fcbe7326b4357a3b2f95aef296e86a2 WatchSource:0}: Error finding container 97b0640e8ef12801914485ee6b21e6410fcbe7326b4357a3b2f95aef296e86a2: Status 404 returned error can't find the container with id 97b0640e8ef12801914485ee6b21e6410fcbe7326b4357a3b2f95aef296e86a2 Jul 22 04:33:09.972597 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.972579 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 04:33:09.986037 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:09.986016 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" Jul 22 04:33:09.995551 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:09.995519 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bb9cb1_2e8a_44c3_8d61_043b7605a224.slice/crio-5c15f4a48d2825fe3e5ffcb3ce19b2fc7cab8ad0e1ec2ccac100b3d639054037 WatchSource:0}: Error finding container 5c15f4a48d2825fe3e5ffcb3ce19b2fc7cab8ad0e1ec2ccac100b3d639054037: Status 404 returned error can't find the container with id 5c15f4a48d2825fe3e5ffcb3ce19b2fc7cab8ad0e1ec2ccac100b3d639054037 Jul 22 04:33:10.006343 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.006325 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" Jul 22 04:33:10.014853 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:10.014831 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fff17b_28b4_43d6_b045_842423eceb38.slice/crio-3d738e22c77c0ad4c1a94b3bbd17e16d259322215cf92bd930fa8ed715722415 WatchSource:0}: Error finding container 3d738e22c77c0ad4c1a94b3bbd17e16d259322215cf92bd930fa8ed715722415: Status 404 returned error can't find the container with id 3d738e22c77c0ad4c1a94b3bbd17e16d259322215cf92bd930fa8ed715722415 Jul 22 04:33:10.015693 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.015677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-drhg7" Jul 22 04:33:10.016345 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.016331 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bmptl" Jul 22 04:33:10.025049 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:10.025028 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb19ef52_a0fa_4b32_8b32_693739754b16.slice/crio-ffb0d1c1602c675b23549bd4bb07a04464d9c873bb0a05385d010936b6908a25 WatchSource:0}: Error finding container ffb0d1c1602c675b23549bd4bb07a04464d9c873bb0a05385d010936b6908a25: Status 404 returned error can't find the container with id ffb0d1c1602c675b23549bd4bb07a04464d9c873bb0a05385d010936b6908a25 Jul 22 04:33:10.028209 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:10.028184 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00b0010_35ef_4ad0_a483_937a188385cc.slice/crio-f681d6989fbcec03c301066b6082b883d9670422b610a2f015d93c2865a5f781 WatchSource:0}: Error finding container f681d6989fbcec03c301066b6082b883d9670422b610a2f015d93c2865a5f781: Status 404 returned error can't find the container with id f681d6989fbcec03c301066b6082b883d9670422b610a2f015d93c2865a5f781 Jul 22 04:33:10.051234 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.051209 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k8rb5" Jul 22 04:33:10.056256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.056234 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rdz55" Jul 22 04:33:10.060559 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:10.060537 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227413da_239d_4d90_8c76_8861f42e0ac0.slice/crio-5f0f9bb6d6466fa0748c6a42f26cf10813b30a5bb35810cfdfa26f82386de87c WatchSource:0}: Error finding container 5f0f9bb6d6466fa0748c6a42f26cf10813b30a5bb35810cfdfa26f82386de87c: Status 404 returned error can't find the container with id 5f0f9bb6d6466fa0748c6a42f26cf10813b30a5bb35810cfdfa26f82386de87c Jul 22 04:33:10.065339 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:10.065316 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbc22f3_78b5_4c69_b089_ea405855ee65.slice/crio-c41e4b1e5c8a5660c1a84d8c41b3f08ec3e5c77418ecfdc23c53582265a9d99f WatchSource:0}: Error finding container c41e4b1e5c8a5660c1a84d8c41b3f08ec3e5c77418ecfdc23c53582265a9d99f: Status 404 returned error can't find the container with id c41e4b1e5c8a5660c1a84d8c41b3f08ec3e5c77418ecfdc23c53582265a9d99f Jul 22 04:33:10.083850 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.083826 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:10.092915 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:10.092892 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6855994a_275d_47be_b3c7_0071685781c0.slice/crio-ff48fda7047c9bc3f2fcbff30464f0bf4708495f0f3f8946125d1108217c7f36 WatchSource:0}: Error finding container ff48fda7047c9bc3f2fcbff30464f0bf4708495f0f3f8946125d1108217c7f36: Status 404 returned error can't find the container with id ff48fda7047c9bc3f2fcbff30464f0bf4708495f0f3f8946125d1108217c7f36 Jul 22 04:33:10.104071 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.104054 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:10.107333 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.107314 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zxf5g" Jul 22 04:33:10.113914 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:10.113894 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66acea06_2cd5_4939_bca5_b63eda7f5b29.slice/crio-537feada3ac709a6685d0344af58ac99e4dfcf62cc72455b4b67264e2790b475 WatchSource:0}: Error finding container 537feada3ac709a6685d0344af58ac99e4dfcf62cc72455b4b67264e2790b475: Status 404 returned error can't find the container with id 537feada3ac709a6685d0344af58ac99e4dfcf62cc72455b4b67264e2790b475 Jul 22 04:33:10.116681 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:10.116662 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63218f68_800c_4699_930f_b228bb95d1f4.slice/crio-dbe429f5860f01abe2ccfa999d9014aa589efdb61178887eb87db303a5a0b97b WatchSource:0}: Error finding container dbe429f5860f01abe2ccfa999d9014aa589efdb61178887eb87db303a5a0b97b: Status 404 returned error can't find the container with id dbe429f5860f01abe2ccfa999d9014aa589efdb61178887eb87db303a5a0b97b Jul 22 04:33:10.119493 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.119472 2572 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-multus start failed in pod multus-zxf5g_openshift-multus(63218f68-800c-4699-930f-b228bb95d1f4): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 22 04:33:10.119549 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.119502 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-zxf5g" podUID="63218f68-800c-4699-930f-b228bb95d1f4" Jul 22 04:33:10.373337 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.373254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:10.373337 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.373299 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:10.373527 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.373412 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:10.373527 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.373469 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret podName:9c051ba5-d2d1-4add-89e0-591295ecaf5e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:11.373453317 +0000 UTC m=+3.092058431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret") pod "global-pull-secret-syncer-hrkws" (UID: "9c051ba5-d2d1-4add-89e0-591295ecaf5e") : object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:10.373527 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.373492 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:10.373637 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.373538 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs podName:0c2de1e1-0496-4eda-b5e5-12b1c181582e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:11.373523194 +0000 UTC m=+3.092128309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs") pod "network-metrics-daemon-wsxd4" (UID: "0c2de1e1-0496-4eda-b5e5-12b1c181582e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:10.473686 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.473648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:10.473886 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.473865 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 04:33:10.473961 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.473895 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 04:33:10.473961 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.473909 2572 projected.go:196] Error preparing data for projected volume kube-api-access-747q7 for pod openshift-network-diagnostics/network-check-target-scr4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:10.474078 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.473969 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7 podName:eda46870-2792-446a-8583-862286c89ab9 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:11.4739508 +0000 UTC m=+3.192555919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-747q7" (UniqueName: "kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7") pod "network-check-target-scr4t" (UID: "eda46870-2792-446a-8583-862286c89ab9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:10.653858 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.653770 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 04:33:10.697483 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.694048 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 04:28:10 +0000 UTC" deadline="2028-04-19 03:15:59.946514722 +0000 UTC" Jul 22 04:33:10.697483 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.694136 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15286h42m49.252383572s" Jul 22 04:33:10.795493 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.795462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:10.795651 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.795600 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:10.795727 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.795689 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:10.795782 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.795763 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:10.834857 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.834792 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k8rb5" event={"ID":"227413da-239d-4d90-8c76-8861f42e0ac0","Type":"ContainerStarted","Data":"5f0f9bb6d6466fa0748c6a42f26cf10813b30a5bb35810cfdfa26f82386de87c"} Jul 22 04:33:10.838876 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.838850 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 04:33:10.840864 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.840836 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmptl" event={"ID":"d00b0010-35ef-4ad0-a483-937a188385cc","Type":"ContainerStarted","Data":"f681d6989fbcec03c301066b6082b883d9670422b610a2f015d93c2865a5f781"} Jul 22 04:33:10.857378 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.857174 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerStarted","Data":"5c15f4a48d2825fe3e5ffcb3ce19b2fc7cab8ad0e1ec2ccac100b3d639054037"} Jul 22 04:33:10.880423 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.880376 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" event={"ID":"480d41ccf0f1f19198559f5c186d8771","Type":"ContainerStarted","Data":"eb81ad100a3d40f45ebd1436716ea7475c65932a0d353845f2466ab01397a820"} Jul 22 04:33:10.900899 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.900857 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" event={"ID":"dec19784c32143c70c3ceb1f215fa236","Type":"ContainerStarted","Data":"97b0640e8ef12801914485ee6b21e6410fcbe7326b4357a3b2f95aef296e86a2"} Jul 22 04:33:10.924234 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.924140 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zxf5g" event={"ID":"63218f68-800c-4699-930f-b228bb95d1f4","Type":"ContainerStarted","Data":"dbe429f5860f01abe2ccfa999d9014aa589efdb61178887eb87db303a5a0b97b"} Jul 22 04:33:10.930943 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.930909 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t5hss" event={"ID":"66acea06-2cd5-4939-bca5-b63eda7f5b29","Type":"ContainerStarted","Data":"537feada3ac709a6685d0344af58ac99e4dfcf62cc72455b4b67264e2790b475"} Jul 22 04:33:10.935777 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:10.935720 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:33e7088dd590651a8add807e644a951c24dc2ea04589a5b08214c6a33ec14009\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-zxf5g" podUID="63218f68-800c-4699-930f-b228bb95d1f4" Jul 22 04:33:10.939276 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.939248 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"ff48fda7047c9bc3f2fcbff30464f0bf4708495f0f3f8946125d1108217c7f36"} Jul 22 04:33:10.953191 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.953146 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-drhg7" event={"ID":"cb19ef52-a0fa-4b32-8b32-693739754b16","Type":"ContainerStarted","Data":"ffb0d1c1602c675b23549bd4bb07a04464d9c873bb0a05385d010936b6908a25"} Jul 22 04:33:10.972406 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.972358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" event={"ID":"a1fff17b-28b4-43d6-b045-842423eceb38","Type":"ContainerStarted","Data":"3d738e22c77c0ad4c1a94b3bbd17e16d259322215cf92bd930fa8ed715722415"} Jul 22 04:33:10.992055 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:10.992016 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rdz55" event={"ID":"7bbc22f3-78b5-4c69-b089-ea405855ee65","Type":"ContainerStarted","Data":"c41e4b1e5c8a5660c1a84d8c41b3f08ec3e5c77418ecfdc23c53582265a9d99f"} Jul 22 04:33:11.238532 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:11.238453 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 04:33:11.380842 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:11.380018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:11.380842 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:11.380072 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:11.380842 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.380250 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:11.380842 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.380317 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs podName:0c2de1e1-0496-4eda-b5e5-12b1c181582e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:13.380294967 +0000 UTC m=+5.098900082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs") pod "network-metrics-daemon-wsxd4" (UID: "0c2de1e1-0496-4eda-b5e5-12b1c181582e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:11.380842 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.380756 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:11.380842 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.380805 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret podName:9c051ba5-d2d1-4add-89e0-591295ecaf5e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:13.380793873 +0000 UTC m=+5.099398988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret") pod "global-pull-secret-syncer-hrkws" (UID: "9c051ba5-d2d1-4add-89e0-591295ecaf5e") : object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:11.480836 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:11.480797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:11.481011 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.480986 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 04:33:11.481011 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.481008 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 04:33:11.481138 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.481023 2572 projected.go:196] Error preparing data for projected volume kube-api-access-747q7 for pod openshift-network-diagnostics/network-check-target-scr4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:11.481138 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.481085 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7 podName:eda46870-2792-446a-8583-862286c89ab9 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:13.481063473 +0000 UTC m=+5.199668592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-747q7" (UniqueName: "kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7") pod "network-check-target-scr4t" (UID: "eda46870-2792-446a-8583-862286c89ab9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:11.694471 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:11.694312 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 04:28:10 +0000 UTC" deadline="2028-03-13 22:00:13.052315934 +0000 UTC" Jul 22 04:33:11.694471 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:11.694381 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14417h27m1.357939266s" Jul 22 04:33:11.794627 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:11.794592 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:11.795016 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:11.794740 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:12.128493 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:12.128401 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 04:33:12.795454 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:12.795422 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:12.795931 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:12.795569 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:12.795931 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:12.795653 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:12.795931 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:12.795727 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:13.397384 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:13.397342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:13.397574 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:13.397421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:13.397659 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.397623 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:13.397711 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.397688 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs podName:0c2de1e1-0496-4eda-b5e5-12b1c181582e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:17.397666082 +0000 UTC m=+9.116271201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs") pod "network-metrics-daemon-wsxd4" (UID: "0c2de1e1-0496-4eda-b5e5-12b1c181582e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:13.398309 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.398286 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:13.398421 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.398352 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret podName:9c051ba5-d2d1-4add-89e0-591295ecaf5e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:17.398330723 +0000 UTC m=+9.116935847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret") pod "global-pull-secret-syncer-hrkws" (UID: "9c051ba5-d2d1-4add-89e0-591295ecaf5e") : object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:13.498845 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:13.498622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:13.498845 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.498817 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 04:33:13.498845 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.498850 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 04:33:13.499151 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.498868 2572 projected.go:196] Error preparing data for projected volume kube-api-access-747q7 for pod openshift-network-diagnostics/network-check-target-scr4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:13.499151 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.498928 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7 podName:eda46870-2792-446a-8583-862286c89ab9 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:17.498910334 +0000 UTC m=+9.217515453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-747q7" (UniqueName: "kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7") pod "network-check-target-scr4t" (UID: "eda46870-2792-446a-8583-862286c89ab9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:13.795147 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:13.795054 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:13.795322 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:13.795202 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:14.795459 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:14.794966 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:14.795459 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:14.795103 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:14.795984 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:14.795610 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:14.795984 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:14.795715 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:15.794966 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:15.794928 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:15.795139 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:15.795069 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:16.795336 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:16.795305 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:16.795823 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:16.795316 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:16.795823 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:16.795456 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:16.795823 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:16.795527 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:17.430054 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:17.430012 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:17.430231 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:17.430065 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:17.430231 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.430194 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:17.430231 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.430216 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:17.430433 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.430276 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret podName:9c051ba5-d2d1-4add-89e0-591295ecaf5e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:25.430255303 +0000 UTC m=+17.148860422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret") pod "global-pull-secret-syncer-hrkws" (UID: "9c051ba5-d2d1-4add-89e0-591295ecaf5e") : object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:17.430433 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.430298 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs podName:0c2de1e1-0496-4eda-b5e5-12b1c181582e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:25.430286901 +0000 UTC m=+17.148892018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs") pod "network-metrics-daemon-wsxd4" (UID: "0c2de1e1-0496-4eda-b5e5-12b1c181582e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:17.530525 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:17.530464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:17.530702 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.530660 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 04:33:17.530702 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.530688 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 04:33:17.530702 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.530703 2572 projected.go:196] Error preparing data for projected volume kube-api-access-747q7 for pod openshift-network-diagnostics/network-check-target-scr4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:17.530868 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.530766 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7 podName:eda46870-2792-446a-8583-862286c89ab9 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:25.530748079 +0000 UTC m=+17.249353213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-747q7" (UniqueName: "kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7") pod "network-check-target-scr4t" (UID: "eda46870-2792-446a-8583-862286c89ab9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:17.795368 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:17.795289 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:17.795752 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:17.795429 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:18.800346 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:18.800275 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:18.800346 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:18.800320 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:18.800855 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:18.800410 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:18.800855 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:18.800524 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:19.794646 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:19.794617 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:19.794820 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:19.794740 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:20.797316 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:20.797294 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:20.797675 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:20.797402 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:20.797675 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:20.797462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:20.797675 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:20.797546 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:21.795642 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:21.795611 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:21.795832 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:21.795761 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:22.795639 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:22.795601 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:22.796102 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:22.795727 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:22.796102 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:22.795794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:22.796102 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:22.795894 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:23.795650 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:23.795616 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:23.796085 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:23.795740 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:24.795489 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:24.795457 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:24.795627 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:24.795457 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:24.795627 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:24.795564 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:24.795917 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:24.795644 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:25.490308 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:25.490090 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:25.490492 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:25.490333 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:25.490492 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.490211 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:25.490594 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.490496 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret podName:9c051ba5-d2d1-4add-89e0-591295ecaf5e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:41.490479094 +0000 UTC m=+33.209084210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret") pod "global-pull-secret-syncer-hrkws" (UID: "9c051ba5-d2d1-4add-89e0-591295ecaf5e") : object "kube-system"/"original-pull-secret" not registered Jul 22 04:33:25.490747 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.490444 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:25.490829 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.490810 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs podName:0c2de1e1-0496-4eda-b5e5-12b1c181582e nodeName:}" failed. No retries permitted until 2026-07-22 04:33:41.490788687 +0000 UTC m=+33.209393811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs") pod "network-metrics-daemon-wsxd4" (UID: "0c2de1e1-0496-4eda-b5e5-12b1c181582e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 04:33:25.591507 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:25.591476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:25.591653 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.591610 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 04:33:25.591653 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.591625 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 04:33:25.591653 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.591636 2572 projected.go:196] Error preparing data for projected volume kube-api-access-747q7 for pod openshift-network-diagnostics/network-check-target-scr4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:25.591784 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.591688 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7 podName:eda46870-2792-446a-8583-862286c89ab9 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:41.591672179 +0000 UTC m=+33.310277312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-747q7" (UniqueName: "kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7") pod "network-check-target-scr4t" (UID: "eda46870-2792-446a-8583-862286c89ab9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 04:33:25.794986 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:25.794857 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:25.795087 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:25.795027 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:26.023677 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.023432 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rdz55" event={"ID":"7bbc22f3-78b5-4c69-b089-ea405855ee65","Type":"ContainerStarted","Data":"94062c4e483154e2503bfea6b8824175c253b7ade69d568b8c960f5b9254c6e2"} Jul 22 04:33:26.025496 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.025458 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k8rb5" event={"ID":"227413da-239d-4d90-8c76-8861f42e0ac0","Type":"ContainerStarted","Data":"2df3541facd8702619d2040d6d830a56acb6e2d3befc3fcc316f118a403d5290"} Jul 22 04:33:26.026985 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.026954 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bmptl" event={"ID":"d00b0010-35ef-4ad0-a483-937a188385cc","Type":"ContainerStarted","Data":"0be13ec5ad0e06669b1dfa93fe72a1d72059755e55bc4fa690b475428d0e53a0"} Jul 22 04:33:26.028469 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.028443 2572 generic.go:356] "Generic (PLEG): container finished" podID="72bb9cb1-2e8a-44c3-8d61-043b7605a224" containerID="ee96b61710f3341ef7df11dded89aca43cd1b2f18e2ede5d1f4242ca93ff40a0" exitCode=0 Jul 22 04:33:26.028564 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.028535 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerDied","Data":"ee96b61710f3341ef7df11dded89aca43cd1b2f18e2ede5d1f4242ca93ff40a0"} Jul 22 04:33:26.030101 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.030079 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" event={"ID":"480d41ccf0f1f19198559f5c186d8771","Type":"ContainerStarted","Data":"02348b69c554269bbe54c868b419d768665263039be6310b3abfc3c10aac7abf"} Jul 22 04:33:26.031960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.031767 2572 generic.go:356] "Generic (PLEG): container finished" podID="dec19784c32143c70c3ceb1f215fa236" containerID="541823cc757797e37eb4e021d8179b937b78a1f44e7a0942f5c3a3d896039f16" exitCode=0 Jul 22 04:33:26.031960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.031857 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" event={"ID":"dec19784c32143c70c3ceb1f215fa236","Type":"ContainerDied","Data":"541823cc757797e37eb4e021d8179b937b78a1f44e7a0942f5c3a3d896039f16"} Jul 22 04:33:26.033513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.033485 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t5hss" event={"ID":"66acea06-2cd5-4939-bca5-b63eda7f5b29","Type":"ContainerStarted","Data":"f574ecee19327c8b9846e9d2bd970ce42bbb4bd44c64b4fb72bb5da326eab50a"} Jul 22 04:33:26.035902 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.035873 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"60a91c016dcb1e00094372e9078786faa2d3c3bfb6b18df8b5a16e540b8f2703"} Jul 22 04:33:26.035996 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.035952 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"bce7dd4a8ee3660a9d6d7b2e3576adccb8ed291bb9f360d2a8d17ea3562c416a"} Jul 22 04:33:26.036082 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.036042 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"318661cb06765832679d4845b030167b122152a52b6a85dc837f88e0b407e21f"} Jul 22 04:33:26.037356 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.037332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" event={"ID":"a1fff17b-28b4-43d6-b045-842423eceb38","Type":"ContainerStarted","Data":"150f793afbd0082b79ff3ccb3b1268e40b487ce2ea7eb7fbe264eb16dec49331"} Jul 22 04:33:26.052792 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.052594 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rdz55" podStartSLOduration=1.745970612 podStartE2EDuration="17.052575848s" totalImagesPullingTime="15.306605236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:26.039239567 +0000 UTC m=+17.757844745" watchObservedRunningTime="2026-07-22 04:33:26.052575848 +0000 UTC m=+17.771180986" Jul 22 04:33:26.074226 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.073585 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t5hss" podStartSLOduration=6.380604014 podStartE2EDuration="17.073567098s" totalImagesPullingTime="10.692963084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:26.052894795 +0000 UTC m=+17.771499931" watchObservedRunningTime="2026-07-22 04:33:26.073567098 +0000 UTC m=+17.792172237" Jul 22 04:33:26.074226 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.073709 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-47.ec2.internal" podStartSLOduration=16.073700587 podStartE2EDuration="16.073700587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:26.072871091 +0000 UTC m=+17.791476227" watchObservedRunningTime="2026-07-22 04:33:26.073700587 +0000 UTC m=+17.792305715" Jul 22 04:33:26.130407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.130338 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bmptl" podStartSLOduration=2.075198184 podStartE2EDuration="17.130317785s" totalImagesPullingTime="15.055119601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:26.129848021 +0000 UTC m=+17.848453157" watchObservedRunningTime="2026-07-22 04:33:26.130317785 +0000 UTC m=+17.848922922" Jul 22 04:33:26.143358 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.143293 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k8rb5" podStartSLOduration=2.118818826 podStartE2EDuration="17.14327669s" totalImagesPullingTime="15.024457864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:26.143003134 +0000 UTC m=+17.861608274" watchObservedRunningTime="2026-07-22 04:33:26.14327669 +0000 UTC m=+17.861881824" Jul 22 04:33:26.794930 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.794718 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:26.795081 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.794763 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:26.795081 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:26.795013 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:26.795180 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:26.795079 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:26.983219 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.983190 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 04:33:26.990183 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.990102 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:26.990631 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:26.990613 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:27.042143 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.042106 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" event={"ID":"dec19784c32143c70c3ceb1f215fa236","Type":"ContainerStarted","Data":"946c3c13d85f272108a6a2e10e120c116d016ad577b6f23c16e1fdc341fbf3b5"} Jul 22 04:33:27.046619 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.046589 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"865fc246ea7f694cde39d65b03e7241cb71c1246bab951fac6f43518ea798dfb"} Jul 22 04:33:27.046722 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.046682 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"7b94101828a957d993f1bcb8540eddb8cf7140789f039bc11e542e14f8927c41"} Jul 22 04:33:27.046775 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.046735 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"a41e644d3a58ee126e056645a546b2a9d4a7c50fea26278f27b56e93bec50068"} Jul 22 04:33:27.048295 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.048270 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-drhg7" event={"ID":"cb19ef52-a0fa-4b32-8b32-693739754b16","Type":"ContainerStarted","Data":"f9a2f987ebed7f8c4fc6ea79d81025394a78ae2911d3809221c336674e034076"} Jul 22 04:33:27.050470 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.050446 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" event={"ID":"a1fff17b-28b4-43d6-b045-842423eceb38","Type":"ContainerStarted","Data":"669378c8356c004663cf6cd039d59c5f1ed2150bbd4143c4b64e7eb0c7291170"} Jul 22 04:33:27.050964 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.050754 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:27.051335 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.051314 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t5hss" Jul 22 04:33:27.056282 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.056244 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-47.ec2.internal" podStartSLOduration=17.056228442 podStartE2EDuration="17.056228442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:27.055863259 +0000 UTC m=+18.774468394" watchObservedRunningTime="2026-07-22 04:33:27.056228442 +0000 UTC m=+18.774833579" Jul 22 04:33:27.071365 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.071318 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-drhg7" podStartSLOduration=3.004801651 podStartE2EDuration="18.071302036s" totalImagesPullingTime="15.066500385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:27.071107461 +0000 UTC m=+18.789712598" watchObservedRunningTime="2026-07-22 04:33:27.071302036 +0000 UTC m=+18.789907172" Jul 22 04:33:27.721024 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.720903 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T04:33:26.98320936Z","UUID":"ef611244-6790-471c-be36-b8162198fac2","Handler":null,"Name":"","Endpoint":""} Jul 22 04:33:27.722882 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.722858 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 04:33:27.723017 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.722900 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 04:33:27.795538 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:27.795510 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:27.795726 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:27.795655 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:28.796027 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:28.795995 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:28.796569 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:28.796106 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:28.796569 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:28.796468 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:28.796694 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:28.796577 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:29.058740 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:29.058655 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"ae1b257348d66aae320041a5a9d47935e4d3db09db8fbcb16b2f87964b2a993b"} Jul 22 04:33:29.061055 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:29.061023 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" event={"ID":"a1fff17b-28b4-43d6-b045-842423eceb38","Type":"ContainerStarted","Data":"6b3b6775ff81557e304bd0b01561f3df73afcf3cb953ec2d69905764db861104"} Jul 22 04:33:29.077809 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:29.077760 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zdf4r" podStartSLOduration=2.9943811399999998 podStartE2EDuration="20.077744501s" totalImagesPullingTime="17.083363361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:29.076170997 +0000 UTC m=+20.794776135" watchObservedRunningTime="2026-07-22 04:33:29.077744501 +0000 UTC m=+20.796349676" Jul 22 04:33:29.795262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:29.795203 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:29.795457 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:29.795346 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:30.799555 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:30.799475 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:30.799984 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:30.799601 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:30.799984 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:30.799649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:30.799984 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:30.799745 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:31.071347 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.070733 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" event={"ID":"6855994a-275d-47be-b3c7-0071685781c0","Type":"ContainerStarted","Data":"4b2a2f0ef844e83101ca49fc8d3be0f45548f6ebcb19d918c642a2fb89d5eb3b"} Jul 22 04:33:31.071347 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.071070 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:31.071347 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.071201 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:31.071347 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.071254 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:31.073421 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.073365 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerStarted","Data":"1d295d5bfa3e501aba003e1290d58f52a593345a662dc8c549cf2e8db3a3a860"} Jul 22 04:33:31.089486 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.089456 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:31.091689 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.091659 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:33:31.099413 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.098871 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" podStartSLOduration=6.492017134 podStartE2EDuration="22.098852114s" totalImagesPullingTime="15.60683498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:31.09730105 +0000 UTC m=+22.815906222" watchObservedRunningTime="2026-07-22 04:33:31.098852114 +0000 UTC m=+22.817457249" Jul 22 04:33:31.794864 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:31.794835 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:31.795109 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:31.794977 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:32.078072 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:32.077990 2572 generic.go:356] "Generic (PLEG): container finished" podID="72bb9cb1-2e8a-44c3-8d61-043b7605a224" containerID="1d295d5bfa3e501aba003e1290d58f52a593345a662dc8c549cf2e8db3a3a860" exitCode=0 Jul 22 04:33:32.079017 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:32.078074 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerDied","Data":"1d295d5bfa3e501aba003e1290d58f52a593345a662dc8c549cf2e8db3a3a860"} Jul 22 04:33:32.799414 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:32.799172 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:32.799578 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:32.799245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:32.799578 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:32.799515 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:32.799674 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:32.799585 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:32.918760 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:32.918724 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrkws"] Jul 22 04:33:32.922039 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:32.921827 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wsxd4"] Jul 22 04:33:32.922039 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:32.921961 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:32.922237 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:32.922088 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:32.923036 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:32.922846 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-scr4t"] Jul 22 04:33:33.080721 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:33.080649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:33.081134 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:33.080659 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:33.081134 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:33.080778 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:33.081134 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:33.080903 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:34.795044 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:34.794965 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:34.795044 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:34.794991 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:34.795537 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:34.795103 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:34.795537 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:34.795170 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:34.795537 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:34.795192 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:34.795537 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:34.795281 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:36.795751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:36.795572 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:36.796167 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:36.795586 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:36.796167 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:36.795838 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-scr4t" podUID="eda46870-2792-446a-8583-862286c89ab9" Jul 22 04:33:36.796167 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:36.795628 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:36.796167 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:36.795909 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wsxd4" podUID="0c2de1e1-0496-4eda-b5e5-12b1c181582e" Jul 22 04:33:36.796167 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:36.796017 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrkws" podUID="9c051ba5-d2d1-4add-89e0-591295ecaf5e" Jul 22 04:33:37.091588 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:37.091552 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zxf5g" event={"ID":"63218f68-800c-4699-930f-b228bb95d1f4","Type":"ContainerStarted","Data":"3c7eb5cb46ce4f6179999b98d4eff6c760ab8ed3cddb8b83031edc4499e1d9f0"} Jul 22 04:33:37.121089 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:37.121024 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zxf5g" podStartSLOduration=1.836169649 podStartE2EDuration="28.121008846s" totalImagesPullingTime="26.284839197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-22 04:33:37.120763529 +0000 UTC m=+28.839368669" watchObservedRunningTime="2026-07-22 04:33:37.121008846 +0000 UTC m=+28.839613982" Jul 22 04:33:38.080283 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.080204 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-47.ec2.internal" event="NodeReady" Jul 22 04:33:38.080788 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.080326 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 04:33:38.095694 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.095670 2572 generic.go:356] "Generic (PLEG): container finished" podID="72bb9cb1-2e8a-44c3-8d61-043b7605a224" containerID="60168a7cb236f216acc01f5a368e207179e95ca18af75a0ba448cfa87d577eb8" exitCode=0 Jul 22 04:33:38.095813 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.095710 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerDied","Data":"60168a7cb236f216acc01f5a368e207179e95ca18af75a0ba448cfa87d577eb8"} Jul 22 04:33:38.110424 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.110397 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79467c9559-6cxq7"] Jul 22 04:33:38.134427 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.134369 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5"] Jul 22 04:33:38.134562 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.134487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.137058 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.137028 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p9jfd\"" Jul 22 04:33:38.137254 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.137242 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 04:33:38.137597 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.137580 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 04:33:38.138636 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.137923 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 04:33:38.144529 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.144510 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 04:33:38.150732 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.150711 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft"] Jul 22 04:33:38.152968 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.152943 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.155717 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.155695 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.155826 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.155739 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 22 04:33:38.155894 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.155845 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-s4mct\"" Jul 22 04:33:38.155989 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.155958 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 22 04:33:38.156124 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.156108 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.179558 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.179537 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct"] Jul 22 04:33:38.179665 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.179646 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft" Jul 22 04:33:38.181917 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.181887 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.181917 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.181903 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-g95rz\"" Jul 22 04:33:38.182065 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.181955 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.189208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.189190 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-image-registry-private-configuration\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.189307 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.189214 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-ca-trust-extracted\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.189307 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.189253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lhc\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-kube-api-access-58lhc\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.189444 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.189307 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.189444 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.189349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-certificates\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.189444 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.189408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-installation-pull-secrets\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.189588 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.189456 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-trusted-ca\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.189588 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.189525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-bound-sa-token\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.199547 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.199528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.199547 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.199545 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-gz2rz"] Jul 22 04:33:38.201437 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.201364 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 22 04:33:38.201720 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.201704 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.201783 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.201757 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.201783 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.201761 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4ldxk\"" Jul 22 04:33:38.201783 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.201761 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 22 04:33:38.217681 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.217658 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d"] Jul 22 04:33:38.217782 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.217767 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.219928 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.219909 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fwtw5\"" Jul 22 04:33:38.219928 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.219909 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 22 04:33:38.220096 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.219928 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.220096 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.219914 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 22 04:33:38.220262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.220250 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.225670 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.225649 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 22 04:33:38.233705 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.233688 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-fkwqf"] Jul 22 04:33:38.233812 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.233798 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:38.235919 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.235898 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2njs6\"" Jul 22 04:33:38.236006 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.235919 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.236152 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.236137 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 04:33:38.236456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.236439 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.257554 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.257527 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm"] Jul 22 04:33:38.257671 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.257655 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.260311 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.260293 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jzh8x\"" Jul 22 04:33:38.260633 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.260617 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 22 04:33:38.260707 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.260669 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.260764 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.260752 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 22 04:33:38.260967 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.260951 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.265243 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.265227 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 22 04:33:38.275106 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.275084 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79467c9559-6cxq7"] Jul 22 04:33:38.275184 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.275113 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft"] Jul 22 04:33:38.275184 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.275128 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5"] Jul 22 04:33:38.275184 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.275140 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm"] Jul 22 04:33:38.275184 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.275152 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d"] Jul 22 04:33:38.275184 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.275171 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h2wp5"] Jul 22 04:33:38.275389 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.275189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.277063 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.277044 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 04:33:38.277127 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.277115 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.277260 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.277247 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.277630 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.277610 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dm9jd\"" Jul 22 04:33:38.277713 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.277649 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 04:33:38.290590 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290571 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lhc\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-kube-api-access-58lhc\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.290673 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290597 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.290673 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-certificates\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.290673 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290634 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:38.290673 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290655 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-installation-pull-secrets\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.290673 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290670 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c59b58-bf2a-40bb-85dd-77a54ced505c-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.290914 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.290722 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 04:33:38.290914 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.290740 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79467c9559-6cxq7: secret "image-registry-tls" not found Jul 22 04:33:38.290914 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.290792 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls podName:a9058cad-5eb5-46fa-89fb-c8ccf93c5b69 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:38.790777948 +0000 UTC m=+30.509383086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls") pod "image-registry-79467c9559-6cxq7" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69") : secret "image-registry-tls" not found Jul 22 04:33:38.290914 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290716 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdmk\" (UniqueName: \"kubernetes.io/projected/18a09dee-af07-4b0a-9df9-ad8b3f6cd049-kube-api-access-vzdmk\") pod \"volume-data-source-validator-78cf4c6f6-zhhft\" (UID: \"18a09dee-af07-4b0a-9df9-ad8b3f6cd049\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft" Jul 22 04:33:38.290914 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290851 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.290914 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd8kk\" (UniqueName: \"kubernetes.io/projected/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-kube-api-access-pd8kk\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.290914 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290907 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6n7t\" (UniqueName: \"kubernetes.io/projected/4696d0cc-1616-4168-a2f8-3bb5319a2073-kube-api-access-n6n7t\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290940 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-trusted-ca\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5c5da81-0039-4b4e-83c4-3f701156340f-config\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.290995 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrzh\" (UniqueName: \"kubernetes.io/projected/d5c5da81-0039-4b4e-83c4-3f701156340f-kube-api-access-fkrzh\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/02c59b58-bf2a-40bb-85dd-77a54ced505c-serving-cert\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291080 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-bound-sa-token\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291116 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c59b58-bf2a-40bb-85dd-77a54ced505c-tmp\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291158 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/02c59b58-bf2a-40bb-85dd-77a54ced505c-snapshots\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291179 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-image-registry-private-configuration\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291195 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-ca-trust-extracted\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291197 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-certificates\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291240 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c5da81-0039-4b4e-83c4-3f701156340f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.291256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291261 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xj9w\" (UniqueName: \"kubernetes.io/projected/02c59b58-bf2a-40bb-85dd-77a54ced505c-kube-api-access-7xj9w\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.291813 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291304 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c59b58-bf2a-40bb-85dd-77a54ced505c-service-ca-bundle\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.291813 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291644 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-ca-trust-extracted\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.291813 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.291660 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-trusted-ca\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.294873 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.294858 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-image-registry-private-configuration\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.294968 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.294950 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-installation-pull-secrets\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.298333 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.298313 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lhc\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-kube-api-access-58lhc\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.298598 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.298574 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-bound-sa-token\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.299357 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.299338 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct"] Jul 22 04:33:38.299458 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.299394 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-fkwqf"] Jul 22 04:33:38.299458 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.299420 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c"] Jul 22 04:33:38.299545 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.299505 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.301507 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.301487 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 04:33:38.301600 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.301584 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 04:33:38.301752 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.301740 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hhmhx\"" Jul 22 04:33:38.321932 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.321909 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh"] Jul 22 04:33:38.322051 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.322031 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:38.324136 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.324109 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 22 04:33:38.324136 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.324113 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h7kj7\"" Jul 22 04:33:38.324290 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.324179 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 22 04:33:38.345998 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.345947 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x5p5s"] Jul 22 04:33:38.346081 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.346060 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh" Jul 22 04:33:38.348074 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.348055 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.348269 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.348253 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.348338 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.348289 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2dqjk\"" Jul 22 04:33:38.370059 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.370034 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-dcfd7dc78-8zf2f"] Jul 22 04:33:38.370148 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.370106 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5p5s" Jul 22 04:33:38.372976 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.372958 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.373198 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.373183 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 04:33:38.373252 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.373237 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.373453 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.373439 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pmc57\"" Jul 22 04:33:38.389462 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.389439 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5p5s"] Jul 22 04:33:38.389558 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.389471 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh"] Jul 22 04:33:38.389558 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.389490 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h2wp5"] Jul 22 04:33:38.389627 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.389554 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.389627 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.389585 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c"] Jul 22 04:33:38.389627 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.389602 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-gz2rz"] Jul 22 04:33:38.389839 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.389741 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-dcfd7dc78-8zf2f"] Jul 22 04:33:38.391511 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391497 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 22 04:33:38.391619 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391602 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 04:33:38.391734 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391646 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 22 04:33:38.391734 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391680 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 22 04:33:38.391869 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391757 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/02c59b58-bf2a-40bb-85dd-77a54ced505c-snapshots\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.391869 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a55ed3ab-0031-410b-bb8b-77c96710507b-trusted-ca\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.391869 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxzcr\" (UniqueName: \"kubernetes.io/projected/a19fb360-7ca0-490a-a7a6-72497334a8ba-kube-api-access-hxzcr\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.391869 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391854 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c5da81-0039-4b4e-83c4-3f701156340f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.392043 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:38.392043 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391912 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px9q4\" (UniqueName: \"kubernetes.io/projected/a55ed3ab-0031-410b-bb8b-77c96710507b-kube-api-access-px9q4\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.392043 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.391951 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:38.392043 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392009 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 04:33:38.392043 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392035 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a55ed3ab-0031-410b-bb8b-77c96710507b-serving-cert\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.392274 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.392098 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 04:33:38.392274 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392112 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.392274 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392139 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sg7t5\"" Jul 22 04:33:38.392274 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392142 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd8kk\" (UniqueName: \"kubernetes.io/projected/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-kube-api-access-pd8kk\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.392274 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392179 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edae6ef2-897e-4f71-8df9-66e34eaaff9c-config-volume\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.392796 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392239 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrzh\" (UniqueName: \"kubernetes.io/projected/d5c5da81-0039-4b4e-83c4-3f701156340f-kube-api-access-fkrzh\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.392796 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/02c59b58-bf2a-40bb-85dd-77a54ced505c-serving-cert\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.392926 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.392814 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls podName:4696d0cc-1616-4168-a2f8-3bb5319a2073 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:38.892781128 +0000 UTC m=+30.611386255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-5bq9d" (UID: "4696d0cc-1616-4168-a2f8-3bb5319a2073") : secret "samples-operator-tls" not found Jul 22 04:33:38.392926 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392888 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.393033 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392935 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.393033 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392976 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/02c59b58-bf2a-40bb-85dd-77a54ced505c-snapshots\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.393033 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.392978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c59b58-bf2a-40bb-85dd-77a54ced505c-tmp\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.393170 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.393043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a55ed3ab-0031-410b-bb8b-77c96710507b-config\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.393170 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.393084 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 22 04:33:38.393170 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.393090 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xj9w\" (UniqueName: \"kubernetes.io/projected/02c59b58-bf2a-40bb-85dd-77a54ced505c-kube-api-access-7xj9w\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.393170 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.393133 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8g7s\" (UniqueName: \"kubernetes.io/projected/edae6ef2-897e-4f71-8df9-66e34eaaff9c-kube-api-access-l8g7s\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.393347 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.393201 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c59b58-bf2a-40bb-85dd-77a54ced505c-service-ca-bundle\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.393347 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.393254 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02c59b58-bf2a-40bb-85dd-77a54ced505c-tmp\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.394076 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.394053 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c59b58-bf2a-40bb-85dd-77a54ced505c-service-ca-bundle\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.394408 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.394386 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdmk\" (UniqueName: \"kubernetes.io/projected/18a09dee-af07-4b0a-9df9-ad8b3f6cd049-kube-api-access-vzdmk\") pod \"volume-data-source-validator-78cf4c6f6-zhhft\" (UID: \"18a09dee-af07-4b0a-9df9-ad8b3f6cd049\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft" Jul 22 04:33:38.395456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.394598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c59b58-bf2a-40bb-85dd-77a54ced505c-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.395456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.394767 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6n7t\" (UniqueName: \"kubernetes.io/projected/4696d0cc-1616-4168-a2f8-3bb5319a2073-kube-api-access-n6n7t\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:38.395456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.394870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5c5da81-0039-4b4e-83c4-3f701156340f-config\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.395456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.395043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.395456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.395081 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a19fb360-7ca0-490a-a7a6-72497334a8ba-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.395456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.395118 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edae6ef2-897e-4f71-8df9-66e34eaaff9c-tmp-dir\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.395456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.395253 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/814f7eac-85c9-4308-a905-c09798acd37c-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:38.396117 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.396098 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c5da81-0039-4b4e-83c4-3f701156340f-serving-cert\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.398071 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.396625 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5c5da81-0039-4b4e-83c4-3f701156340f-config\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.398071 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.396987 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/02c59b58-bf2a-40bb-85dd-77a54ced505c-serving-cert\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.398071 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.397870 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02c59b58-bf2a-40bb-85dd-77a54ced505c-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.399356 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.399312 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.404167 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.404146 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd8kk\" (UniqueName: \"kubernetes.io/projected/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-kube-api-access-pd8kk\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.404327 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.404305 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrzh\" (UniqueName: \"kubernetes.io/projected/d5c5da81-0039-4b4e-83c4-3f701156340f-kube-api-access-fkrzh\") pod \"service-ca-operator-5bcbc94c78-sdlct\" (UID: \"d5c5da81-0039-4b4e-83c4-3f701156340f\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.404569 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.404553 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xj9w\" (UniqueName: \"kubernetes.io/projected/02c59b58-bf2a-40bb-85dd-77a54ced505c-kube-api-access-7xj9w\") pod \"insights-operator-555d9d7885-gz2rz\" (UID: \"02c59b58-bf2a-40bb-85dd-77a54ced505c\") " pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.404874 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.404851 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6n7t\" (UniqueName: \"kubernetes.io/projected/4696d0cc-1616-4168-a2f8-3bb5319a2073-kube-api-access-n6n7t\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:38.404993 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.404978 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdmk\" (UniqueName: \"kubernetes.io/projected/18a09dee-af07-4b0a-9df9-ad8b3f6cd049-kube-api-access-vzdmk\") pod \"volume-data-source-validator-78cf4c6f6-zhhft\" (UID: \"18a09dee-af07-4b0a-9df9-ad8b3f6cd049\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft" Jul 22 04:33:38.407528 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.407511 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e47c0c3-83ee-4f91-b30d-83203d3d2bb5-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mrht5\" (UID: \"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.461991 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.461964 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" Jul 22 04:33:38.488009 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.487986 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft" Jul 22 04:33:38.496557 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496531 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvl9\" (UniqueName: \"kubernetes.io/projected/e246d46d-6258-4941-a2c0-c71e777bf0ad-kube-api-access-slvl9\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.496658 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496612 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.496714 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a19fb360-7ca0-490a-a7a6-72497334a8ba-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.496751 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edae6ef2-897e-4f71-8df9-66e34eaaff9c-tmp-dir\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.496799 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496749 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/814f7eac-85c9-4308-a905-c09798acd37c-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:38.496799 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496785 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.496896 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a55ed3ab-0031-410b-bb8b-77c96710507b-trusted-ca\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.496896 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496852 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxzcr\" (UniqueName: \"kubernetes.io/projected/a19fb360-7ca0-490a-a7a6-72497334a8ba-kube-api-access-hxzcr\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.496896 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:38.497027 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496910 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfqd\" (UniqueName: \"kubernetes.io/projected/2f4adc27-ec9e-47f4-8086-9e7b0104e4c4-kube-api-access-7cfqd\") pod \"network-check-source-7bc5b6bf85-mmslh\" (UID: \"2f4adc27-ec9e-47f4-8086-9e7b0104e4c4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh" Jul 22 04:33:38.497027 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-stats-auth\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.497027 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.496962 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px9q4\" (UniqueName: \"kubernetes.io/projected/a55ed3ab-0031-410b-bb8b-77c96710507b-kube-api-access-px9q4\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.497027 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a55ed3ab-0031-410b-bb8b-77c96710507b-serving-cert\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.497154 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497031 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48e78c79-6a2d-44cc-8f84-c123315037b7-cert\") pod \"ingress-canary-x5p5s\" (UID: \"48e78c79-6a2d-44cc-8f84-c123315037b7\") " pod="openshift-ingress-canary/ingress-canary-x5p5s" Jul 22 04:33:38.497154 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbs7\" (UniqueName: \"kubernetes.io/projected/48e78c79-6a2d-44cc-8f84-c123315037b7-kube-api-access-8dbs7\") pod \"ingress-canary-x5p5s\" (UID: \"48e78c79-6a2d-44cc-8f84-c123315037b7\") " pod="openshift-ingress-canary/ingress-canary-x5p5s" Jul 22 04:33:38.497154 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497081 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-default-certificate\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.497154 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497094 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edae6ef2-897e-4f71-8df9-66e34eaaff9c-tmp-dir\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.497610 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497587 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a19fb360-7ca0-490a-a7a6-72497334a8ba-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.497698 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.497683 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 04:33:38.497763 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.497754 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert podName:814f7eac-85c9-4308-a905-c09798acd37c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:38.997735863 +0000 UTC m=+30.716340978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4qc7c" (UID: "814f7eac-85c9-4308-a905-c09798acd37c") : secret "networking-console-plugin-cert" not found Jul 22 04:33:38.497881 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edae6ef2-897e-4f71-8df9-66e34eaaff9c-config-volume\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.497926 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497910 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.497956 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497944 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.497993 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.497980 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a55ed3ab-0031-410b-bb8b-77c96710507b-config\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.498038 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.498023 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8g7s\" (UniqueName: \"kubernetes.io/projected/edae6ef2-897e-4f71-8df9-66e34eaaff9c-kube-api-access-l8g7s\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.498507 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.498127 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a55ed3ab-0031-410b-bb8b-77c96710507b-trusted-ca\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.498507 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.498341 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:38.498507 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.498417 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls podName:a19fb360-7ca0-490a-a7a6-72497334a8ba nodeName:}" failed. No retries permitted until 2026-07-22 04:33:38.99839977 +0000 UTC m=+30.717004898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8lqlm" (UID: "a19fb360-7ca0-490a-a7a6-72497334a8ba") : secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:38.498507 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.498427 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/814f7eac-85c9-4308-a905-c09798acd37c-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:38.498507 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.498472 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edae6ef2-897e-4f71-8df9-66e34eaaff9c-config-volume\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.498507 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.498480 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 04:33:38.498876 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.498532 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls podName:edae6ef2-897e-4f71-8df9-66e34eaaff9c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:38.998519951 +0000 UTC m=+30.717125065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls") pod "dns-default-h2wp5" (UID: "edae6ef2-897e-4f71-8df9-66e34eaaff9c") : secret "dns-default-metrics-tls" not found Jul 22 04:33:38.498954 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.498932 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a55ed3ab-0031-410b-bb8b-77c96710507b-config\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.500201 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.500184 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a55ed3ab-0031-410b-bb8b-77c96710507b-serving-cert\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.506261 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.506223 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8g7s\" (UniqueName: \"kubernetes.io/projected/edae6ef2-897e-4f71-8df9-66e34eaaff9c-kube-api-access-l8g7s\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:38.506620 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.506576 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxzcr\" (UniqueName: \"kubernetes.io/projected/a19fb360-7ca0-490a-a7a6-72497334a8ba-kube-api-access-hxzcr\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:38.506712 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.506658 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px9q4\" (UniqueName: \"kubernetes.io/projected/a55ed3ab-0031-410b-bb8b-77c96710507b-kube-api-access-px9q4\") pod \"console-operator-565b7b569d-fkwqf\" (UID: \"a55ed3ab-0031-410b-bb8b-77c96710507b\") " pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.507206 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.507182 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" Jul 22 04:33:38.526858 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.526832 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-gz2rz" Jul 22 04:33:38.566321 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.565975 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:38.598798 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.598661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.598960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.599043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfqd\" (UniqueName: \"kubernetes.io/projected/2f4adc27-ec9e-47f4-8086-9e7b0104e4c4-kube-api-access-7cfqd\") pod \"network-check-source-7bc5b6bf85-mmslh\" (UID: \"2f4adc27-ec9e-47f4-8086-9e7b0104e4c4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh" Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.599070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-stats-auth\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.599118 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48e78c79-6a2d-44cc-8f84-c123315037b7-cert\") pod \"ingress-canary-x5p5s\" (UID: \"48e78c79-6a2d-44cc-8f84-c123315037b7\") " pod="openshift-ingress-canary/ingress-canary-x5p5s" Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.599143 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbs7\" (UniqueName: \"kubernetes.io/projected/48e78c79-6a2d-44cc-8f84-c123315037b7-kube-api-access-8dbs7\") pod \"ingress-canary-x5p5s\" (UID: \"48e78c79-6a2d-44cc-8f84-c123315037b7\") " pod="openshift-ingress-canary/ingress-canary-x5p5s" Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.599167 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-default-certificate\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.599265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvl9\" (UniqueName: \"kubernetes.io/projected/e246d46d-6258-4941-a2c0-c71e777bf0ad-kube-api-access-slvl9\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.600001 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:39.099977414 +0000 UTC m=+30.818582541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : configmap references non-existent config key: service-ca.crt Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.600219 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 04:33:38.600763 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.600257 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:39.100244003 +0000 UTC m=+30.818849120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : secret "router-metrics-certs-default" not found Jul 22 04:33:38.607541 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.607491 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48e78c79-6a2d-44cc-8f84-c123315037b7-cert\") pod \"ingress-canary-x5p5s\" (UID: \"48e78c79-6a2d-44cc-8f84-c123315037b7\") " pod="openshift-ingress-canary/ingress-canary-x5p5s" Jul 22 04:33:38.610108 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.610056 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-stats-auth\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.626134 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.626065 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-default-certificate\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.630266 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.630211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfqd\" (UniqueName: \"kubernetes.io/projected/2f4adc27-ec9e-47f4-8086-9e7b0104e4c4-kube-api-access-7cfqd\") pod \"network-check-source-7bc5b6bf85-mmslh\" (UID: \"2f4adc27-ec9e-47f4-8086-9e7b0104e4c4\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh" Jul 22 04:33:38.630409 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.630272 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvl9\" (UniqueName: \"kubernetes.io/projected/e246d46d-6258-4941-a2c0-c71e777bf0ad-kube-api-access-slvl9\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:38.635884 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.635784 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbs7\" (UniqueName: \"kubernetes.io/projected/48e78c79-6a2d-44cc-8f84-c123315037b7-kube-api-access-8dbs7\") pod \"ingress-canary-x5p5s\" (UID: \"48e78c79-6a2d-44cc-8f84-c123315037b7\") " pod="openshift-ingress-canary/ingress-canary-x5p5s" Jul 22 04:33:38.654414 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.654125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh" Jul 22 04:33:38.678647 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.678577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x5p5s" Jul 22 04:33:38.689911 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.689716 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5"] Jul 22 04:33:38.695131 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:38.694906 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e47c0c3_83ee_4f91_b30d_83203d3d2bb5.slice/crio-34f6d040b9934848058d3b77a971ac2df8f8cc60849056f697c913f609f944e9 WatchSource:0}: Error finding container 34f6d040b9934848058d3b77a971ac2df8f8cc60849056f697c913f609f944e9: Status 404 returned error can't find the container with id 34f6d040b9934848058d3b77a971ac2df8f8cc60849056f697c913f609f944e9 Jul 22 04:33:38.739766 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.739716 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft"] Jul 22 04:33:38.743598 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.743216 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct"] Jul 22 04:33:38.771311 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.771219 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-gz2rz"] Jul 22 04:33:38.797025 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.795443 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:38.800552 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.797830 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:38.800552 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.798235 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:38.802556 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.801947 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 04:33:38.802556 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.801975 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j88w2\"" Jul 22 04:33:38.802556 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.802195 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 04:33:38.802556 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.802350 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ljtjg\"" Jul 22 04:33:38.803018 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.802944 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:38.803187 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.803168 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 04:33:38.803261 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.803191 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79467c9559-6cxq7: secret "image-registry-tls" not found Jul 22 04:33:38.803261 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.803244 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls podName:a9058cad-5eb5-46fa-89fb-c8ccf93c5b69 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:39.80322688 +0000 UTC m=+31.521832006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls") pod "image-registry-79467c9559-6cxq7" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69") : secret "image-registry-tls" not found Jul 22 04:33:38.806195 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.806137 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-fkwqf"] Jul 22 04:33:38.856191 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.856121 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh"] Jul 22 04:33:38.878142 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.878110 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x5p5s"] Jul 22 04:33:38.904322 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:38.904296 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:38.904483 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.904451 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 04:33:38.904539 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:38.904507 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls podName:4696d0cc-1616-4168-a2f8-3bb5319a2073 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:39.904490234 +0000 UTC m=+31.623095347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-5bq9d" (UID: "4696d0cc-1616-4168-a2f8-3bb5319a2073") : secret "samples-operator-tls" not found Jul 22 04:33:38.923833 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:38.923809 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c59b58_bf2a_40bb_85dd_77a54ced505c.slice/crio-270d068505c227c3ca9fb06e2b1c15ed39db16ef2a7f9640dcb45e01ac28729a WatchSource:0}: Error finding container 270d068505c227c3ca9fb06e2b1c15ed39db16ef2a7f9640dcb45e01ac28729a: Status 404 returned error can't find the container with id 270d068505c227c3ca9fb06e2b1c15ed39db16ef2a7f9640dcb45e01ac28729a Jul 22 04:33:38.924403 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:38.924353 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4adc27_ec9e_47f4_8086_9e7b0104e4c4.slice/crio-3ef38f4f4f6308a593c5ea0ababa2ec90a9a9a749ac0348aae31203194714665 WatchSource:0}: Error finding container 3ef38f4f4f6308a593c5ea0ababa2ec90a9a9a749ac0348aae31203194714665: Status 404 returned error can't find the container with id 3ef38f4f4f6308a593c5ea0ababa2ec90a9a9a749ac0348aae31203194714665 Jul 22 04:33:38.925421 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:38.925367 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c5da81_0039_4b4e_83c4_3f701156340f.slice/crio-e8d10ee916e3113b071437aae51c1533da3975dab732a353e24bc6669b13693d WatchSource:0}: Error finding container e8d10ee916e3113b071437aae51c1533da3975dab732a353e24bc6669b13693d: Status 404 returned error can't find the container with id e8d10ee916e3113b071437aae51c1533da3975dab732a353e24bc6669b13693d Jul 22 04:33:39.005559 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.005529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:39.005697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.005601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:39.005697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.005646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:39.005697 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.005690 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 04:33:39.005806 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.005748 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 04:33:39.005806 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.005749 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:39.005806 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.005754 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert podName:814f7eac-85c9-4308-a905-c09798acd37c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:40.00573244 +0000 UTC m=+31.724337576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4qc7c" (UID: "814f7eac-85c9-4308-a905-c09798acd37c") : secret "networking-console-plugin-cert" not found Jul 22 04:33:39.005907 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.005813 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls podName:a19fb360-7ca0-490a-a7a6-72497334a8ba nodeName:}" failed. No retries permitted until 2026-07-22 04:33:40.005787637 +0000 UTC m=+31.724392749 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8lqlm" (UID: "a19fb360-7ca0-490a-a7a6-72497334a8ba") : secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:39.005907 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.005838 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls podName:edae6ef2-897e-4f71-8df9-66e34eaaff9c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:40.005821277 +0000 UTC m=+31.724426397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls") pod "dns-default-h2wp5" (UID: "edae6ef2-897e-4f71-8df9-66e34eaaff9c") : secret "dns-default-metrics-tls" not found Jul 22 04:33:39.099625 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.099593 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" event={"ID":"d5c5da81-0039-4b4e-83c4-3f701156340f","Type":"ContainerStarted","Data":"e8d10ee916e3113b071437aae51c1533da3975dab732a353e24bc6669b13693d"} Jul 22 04:33:39.100678 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.100656 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft" event={"ID":"18a09dee-af07-4b0a-9df9-ad8b3f6cd049","Type":"ContainerStarted","Data":"2da7bf5f43e798a720129c1aba2cd58f552cc0af64f0e1fab39b9f98efdb3c4b"} Jul 22 04:33:39.101584 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.101562 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-gz2rz" event={"ID":"02c59b58-bf2a-40bb-85dd-77a54ced505c","Type":"ContainerStarted","Data":"270d068505c227c3ca9fb06e2b1c15ed39db16ef2a7f9640dcb45e01ac28729a"} Jul 22 04:33:39.102576 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.102540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh" event={"ID":"2f4adc27-ec9e-47f4-8086-9e7b0104e4c4","Type":"ContainerStarted","Data":"3ef38f4f4f6308a593c5ea0ababa2ec90a9a9a749ac0348aae31203194714665"} Jul 22 04:33:39.103494 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.103468 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" event={"ID":"a55ed3ab-0031-410b-bb8b-77c96710507b","Type":"ContainerStarted","Data":"d221d1d3df94386e5e7cc53c936dae1834485a758b031fce8835a7dc97c755dd"} Jul 22 04:33:39.104428 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.104395 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" event={"ID":"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5","Type":"ContainerStarted","Data":"34f6d040b9934848058d3b77a971ac2df8f8cc60849056f697c913f609f944e9"} Jul 22 04:33:39.105483 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.105461 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5p5s" event={"ID":"48e78c79-6a2d-44cc-8f84-c123315037b7","Type":"ContainerStarted","Data":"5a194248fb2cde84de442638006e0e8c1a7b842ee957114753ef0470ae556896"} Jul 22 04:33:39.106859 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.106816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:39.106859 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.106846 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:39.107032 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.106960 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:40.106938165 +0000 UTC m=+31.825543286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : configmap references non-existent config key: service-ca.crt Jul 22 04:33:39.107032 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.107013 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 04:33:39.107099 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.107058 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:40.107045669 +0000 UTC m=+31.825650803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : secret "router-metrics-certs-default" not found Jul 22 04:33:39.815786 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.815065 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:39.815786 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.815273 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 04:33:39.815786 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.815289 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79467c9559-6cxq7: secret "image-registry-tls" not found Jul 22 04:33:39.815786 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.815347 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls podName:a9058cad-5eb5-46fa-89fb-c8ccf93c5b69 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:41.815328584 +0000 UTC m=+33.533933703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls") pod "image-registry-79467c9559-6cxq7" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69") : secret "image-registry-tls" not found Jul 22 04:33:39.916412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:39.916357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:39.916656 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.916636 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 04:33:39.916738 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:39.916709 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls podName:4696d0cc-1616-4168-a2f8-3bb5319a2073 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:41.916689002 +0000 UTC m=+33.635294129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-5bq9d" (UID: "4696d0cc-1616-4168-a2f8-3bb5319a2073") : secret "samples-operator-tls" not found Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:40.017701 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:40.017799 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:40.017828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.017883 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.017944 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.017960 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert podName:814f7eac-85c9-4308-a905-c09798acd37c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:42.017939007 +0000 UTC m=+33.736544126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4qc7c" (UID: "814f7eac-85c9-4308-a905-c09798acd37c") : secret "networking-console-plugin-cert" not found Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.017999 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls podName:a19fb360-7ca0-490a-a7a6-72497334a8ba nodeName:}" failed. No retries permitted until 2026-07-22 04:33:42.01798238 +0000 UTC m=+33.736587493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8lqlm" (UID: "a19fb360-7ca0-490a-a7a6-72497334a8ba") : secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.018059 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 04:33:40.018111 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.018089 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls podName:edae6ef2-897e-4f71-8df9-66e34eaaff9c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:42.018078497 +0000 UTC m=+33.736683614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls") pod "dns-default-h2wp5" (UID: "edae6ef2-897e-4f71-8df9-66e34eaaff9c") : secret "dns-default-metrics-tls" not found Jul 22 04:33:40.117771 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:40.116668 2572 generic.go:356] "Generic (PLEG): container finished" podID="72bb9cb1-2e8a-44c3-8d61-043b7605a224" containerID="1fd5b4e5adff5187e94c3c2b328ab4f2defadadea99c63d72f0b47a7bf4dc20d" exitCode=0 Jul 22 04:33:40.117771 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:40.116866 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerDied","Data":"1fd5b4e5adff5187e94c3c2b328ab4f2defadadea99c63d72f0b47a7bf4dc20d"} Jul 22 04:33:40.118300 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:40.118243 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:40.118300 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:40.118286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:40.119017 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.118532 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 04:33:40.119017 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.118590 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:42.118571652 +0000 UTC m=+33.837176771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : secret "router-metrics-certs-default" not found Jul 22 04:33:40.119017 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:40.118960 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:42.118944343 +0000 UTC m=+33.837549461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : configmap references non-existent config key: service-ca.crt Jul 22 04:33:41.534941 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.533613 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:41.534941 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.533665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:33:41.534941 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:41.533874 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 22 04:33:41.534941 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:41.533929 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs podName:0c2de1e1-0496-4eda-b5e5-12b1c181582e nodeName:}" failed. No retries permitted until 2026-07-22 04:34:13.533910925 +0000 UTC m=+65.252516039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs") pod "network-metrics-daemon-wsxd4" (UID: "0c2de1e1-0496-4eda-b5e5-12b1c181582e") : secret "metrics-daemon-secret" not found Jul 22 04:33:41.539607 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.539577 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9c051ba5-d2d1-4add-89e0-591295ecaf5e-original-pull-secret\") pod \"global-pull-secret-syncer-hrkws\" (UID: \"9c051ba5-d2d1-4add-89e0-591295ecaf5e\") " pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:41.634617 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.634579 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:41.639721 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.639658 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-747q7\" (UniqueName: \"kubernetes.io/projected/eda46870-2792-446a-8583-862286c89ab9-kube-api-access-747q7\") pod \"network-check-target-scr4t\" (UID: \"eda46870-2792-446a-8583-862286c89ab9\") " pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:41.827158 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.827073 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:41.834565 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.834488 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrkws" Jul 22 04:33:41.837097 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.837069 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:41.837274 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:41.837252 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 04:33:41.837343 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:41.837278 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79467c9559-6cxq7: secret "image-registry-tls" not found Jul 22 04:33:41.837410 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:41.837344 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls podName:a9058cad-5eb5-46fa-89fb-c8ccf93c5b69 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:45.837323642 +0000 UTC m=+37.555928756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls") pod "image-registry-79467c9559-6cxq7" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69") : secret "image-registry-tls" not found Jul 22 04:33:41.937816 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:41.937699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:41.937978 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:41.937854 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 04:33:41.937978 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:41.937932 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls podName:4696d0cc-1616-4168-a2f8-3bb5319a2073 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:45.937904232 +0000 UTC m=+37.656509355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-5bq9d" (UID: "4696d0cc-1616-4168-a2f8-3bb5319a2073") : secret "samples-operator-tls" not found Jul 22 04:33:42.039220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:42.039179 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:42.039406 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:42.039283 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:42.039406 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:42.039313 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:42.039406 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.039321 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 04:33:42.039585 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.039421 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert podName:814f7eac-85c9-4308-a905-c09798acd37c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:46.039396597 +0000 UTC m=+37.758001726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4qc7c" (UID: "814f7eac-85c9-4308-a905-c09798acd37c") : secret "networking-console-plugin-cert" not found Jul 22 04:33:42.039585 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.039464 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 04:33:42.039585 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.039468 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:42.039585 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.039522 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls podName:edae6ef2-897e-4f71-8df9-66e34eaaff9c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:46.039504171 +0000 UTC m=+37.758109289 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls") pod "dns-default-h2wp5" (UID: "edae6ef2-897e-4f71-8df9-66e34eaaff9c") : secret "dns-default-metrics-tls" not found Jul 22 04:33:42.039585 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.039577 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls podName:a19fb360-7ca0-490a-a7a6-72497334a8ba nodeName:}" failed. No retries permitted until 2026-07-22 04:33:46.039565156 +0000 UTC m=+37.758170269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8lqlm" (UID: "a19fb360-7ca0-490a-a7a6-72497334a8ba") : secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:42.140361 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:42.140271 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:42.140361 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:42.140334 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:42.140586 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.140479 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 04:33:42.140586 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.140495 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:46.140471123 +0000 UTC m=+37.859076239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : configmap references non-existent config key: service-ca.crt Jul 22 04:33:42.140586 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:42.140540 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:46.140523501 +0000 UTC m=+37.859128619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : secret "router-metrics-certs-default" not found Jul 22 04:33:45.876707 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:45.876661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:45.877354 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:45.876828 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 04:33:45.877354 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:45.876852 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79467c9559-6cxq7: secret "image-registry-tls" not found Jul 22 04:33:45.877354 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:45.876913 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls podName:a9058cad-5eb5-46fa-89fb-c8ccf93c5b69 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:53.876893687 +0000 UTC m=+45.595498799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls") pod "image-registry-79467c9559-6cxq7" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69") : secret "image-registry-tls" not found Jul 22 04:33:45.977974 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:45.977927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:45.978148 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:45.978095 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 04:33:45.978206 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:45.978171 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls podName:4696d0cc-1616-4168-a2f8-3bb5319a2073 nodeName:}" failed. No retries permitted until 2026-07-22 04:33:53.978151185 +0000 UTC m=+45.696756300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-5bq9d" (UID: "4696d0cc-1616-4168-a2f8-3bb5319a2073") : secret "samples-operator-tls" not found Jul 22 04:33:46.078579 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:46.078537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:46.078755 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:46.078625 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:46.078755 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:46.078656 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:46.078755 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.078679 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 04:33:46.078755 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.078748 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert podName:814f7eac-85c9-4308-a905-c09798acd37c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:54.078732353 +0000 UTC m=+45.797337466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4qc7c" (UID: "814f7eac-85c9-4308-a905-c09798acd37c") : secret "networking-console-plugin-cert" not found Jul 22 04:33:46.078931 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.078783 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:46.078931 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.078786 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 04:33:46.078931 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.078833 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls podName:a19fb360-7ca0-490a-a7a6-72497334a8ba nodeName:}" failed. No retries permitted until 2026-07-22 04:33:54.078821577 +0000 UTC m=+45.797426695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8lqlm" (UID: "a19fb360-7ca0-490a-a7a6-72497334a8ba") : secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:46.078931 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.078850 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls podName:edae6ef2-897e-4f71-8df9-66e34eaaff9c nodeName:}" failed. No retries permitted until 2026-07-22 04:33:54.078840389 +0000 UTC m=+45.797445502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls") pod "dns-default-h2wp5" (UID: "edae6ef2-897e-4f71-8df9-66e34eaaff9c") : secret "dns-default-metrics-tls" not found Jul 22 04:33:46.179644 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:46.179556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:46.179644 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:46.179611 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:46.179871 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.179753 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:54.17972989 +0000 UTC m=+45.898335007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : configmap references non-existent config key: service-ca.crt Jul 22 04:33:46.179871 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.179784 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 04:33:46.179871 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:46.179852 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:33:54.179834955 +0000 UTC m=+45.898440086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : secret "router-metrics-certs-default" not found Jul 22 04:33:48.813782 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:48.813747 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-scr4t"] Jul 22 04:33:48.814335 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:48.814320 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrkws"] Jul 22 04:33:48.816331 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:33:48.816298 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c051ba5_d2d1_4add_89e0_591295ecaf5e.slice/crio-e8272b44dfa8af8b034ffcb21d2aa0024e103a7fc716836dcce06a9dfb53696e WatchSource:0}: Error finding container e8272b44dfa8af8b034ffcb21d2aa0024e103a7fc716836dcce06a9dfb53696e: Status 404 returned error can't find the container with id e8272b44dfa8af8b034ffcb21d2aa0024e103a7fc716836dcce06a9dfb53696e Jul 22 04:33:49.143982 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.143949 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-gz2rz" event={"ID":"02c59b58-bf2a-40bb-85dd-77a54ced505c","Type":"ContainerStarted","Data":"fbd1f876001b08c3c1d097be3b790bc8cc919ff568a5d8596abb84234cd1e1d6"} Jul 22 04:33:49.145684 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.145634 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-scr4t" event={"ID":"eda46870-2792-446a-8583-862286c89ab9","Type":"ContainerStarted","Data":"767464b9c7e01a0580fedde9046e6cbb12109b8a65cbef7570f9ff7cafc09216"} Jul 22 04:33:49.145684 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.145680 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-scr4t" event={"ID":"eda46870-2792-446a-8583-862286c89ab9","Type":"ContainerStarted","Data":"2c17c6679f0de4f91432c51c3458555e59311df82e61181e27c4b74a6cc5d619"} Jul 22 04:33:49.145847 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.145738 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:33:49.147466 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.147426 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh" event={"ID":"2f4adc27-ec9e-47f4-8086-9e7b0104e4c4","Type":"ContainerStarted","Data":"7d1103dbdfe77cdb082d06989ac08a81d5cdd2c51abf8097b326925b73206f24"} Jul 22 04:33:49.149184 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.149167 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/0.log" Jul 22 04:33:49.149275 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.149206 2572 generic.go:356] "Generic (PLEG): container finished" podID="a55ed3ab-0031-410b-bb8b-77c96710507b" containerID="c953e142e05f7c20b902c3336f5c408aa08ac342f75e0397f8e7a77c0855a3eb" exitCode=255 Jul 22 04:33:49.149332 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.149283 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" event={"ID":"a55ed3ab-0031-410b-bb8b-77c96710507b","Type":"ContainerDied","Data":"c953e142e05f7c20b902c3336f5c408aa08ac342f75e0397f8e7a77c0855a3eb"} Jul 22 04:33:49.149460 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.149444 2572 scope.go:117] "RemoveContainer" containerID="c953e142e05f7c20b902c3336f5c408aa08ac342f75e0397f8e7a77c0855a3eb" Jul 22 04:33:49.151101 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.151079 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" event={"ID":"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5","Type":"ContainerStarted","Data":"fe23de0a9046168747df02f87ab544e05e2f43e592e5be28cec85d4e04729ba2"} Jul 22 04:33:49.156098 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.156067 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrkws" event={"ID":"9c051ba5-d2d1-4add-89e0-591295ecaf5e","Type":"ContainerStarted","Data":"e8272b44dfa8af8b034ffcb21d2aa0024e103a7fc716836dcce06a9dfb53696e"} Jul 22 04:33:49.157669 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.157645 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x5p5s" event={"ID":"48e78c79-6a2d-44cc-8f84-c123315037b7","Type":"ContainerStarted","Data":"c5d37e9cb6b939e399e93d5c2612c7e128beedff8477a736876674acd70c7727"} Jul 22 04:33:49.161728 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.161705 2572 generic.go:356] "Generic (PLEG): container finished" podID="72bb9cb1-2e8a-44c3-8d61-043b7605a224" containerID="365f8feed43b3d4acb6b15cb829ecb65d5644d424b253414187f6753a49369e9" exitCode=0 Jul 22 04:33:49.161826 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.161778 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerDied","Data":"365f8feed43b3d4acb6b15cb829ecb65d5644d424b253414187f6753a49369e9"} Jul 22 04:33:49.163343 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.163288 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" event={"ID":"d5c5da81-0039-4b4e-83c4-3f701156340f","Type":"ContainerStarted","Data":"65632ca85e5d23562515718572f8f6522d68822101f62401c6bd9e12a3760a11"} Jul 22 04:33:49.165397 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.165054 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft" event={"ID":"18a09dee-af07-4b0a-9df9-ad8b3f6cd049","Type":"ContainerStarted","Data":"98c38ada971a017fe03cd86507a1f2f24f4cd0ebdb94e17785ca3b03d1e55ac3"} Jul 22 04:33:49.173630 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.172088 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-scr4t" podStartSLOduration=40.172072503 podStartE2EDuration="40.172072503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:49.171615981 +0000 UTC m=+40.890221117" watchObservedRunningTime="2026-07-22 04:33:49.172072503 +0000 UTC m=+40.890677639" Jul 22 04:33:49.173630 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.173064 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-gz2rz" podStartSLOduration=10.452764336 podStartE2EDuration="20.173046925s" totalImagesPullingTime="9.720282589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:49.158357274 +0000 UTC m=+40.876962410" watchObservedRunningTime="2026-07-22 04:33:49.173046925 +0000 UTC m=+40.891652063" Jul 22 04:33:49.191582 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.189239 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" podStartSLOduration=10.322546827 podStartE2EDuration="20.189225561s" totalImagesPullingTime="9.866678734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:49.18901382 +0000 UTC m=+40.907618956" watchObservedRunningTime="2026-07-22 04:33:49.189225561 +0000 UTC m=+40.907830677" Jul 22 04:33:49.230030 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.229985 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-mmslh" podStartSLOduration=10.508193308 podStartE2EDuration="20.229963796s" totalImagesPullingTime="9.721770488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:49.228863103 +0000 UTC m=+40.947468231" watchObservedRunningTime="2026-07-22 04:33:49.229963796 +0000 UTC m=+40.948568932" Jul 22 04:33:49.245349 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.245292 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x5p5s" podStartSLOduration=1.527900544 podStartE2EDuration="11.245273799s" totalImagesPullingTime="9.717373255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:49.244448757 +0000 UTC m=+40.963053893" watchObservedRunningTime="2026-07-22 04:33:49.245273799 +0000 UTC m=+40.963878935" Jul 22 04:33:49.282311 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.282263 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-zhhft" podStartSLOduration=11.085449584 podStartE2EDuration="20.28224433s" totalImagesPullingTime="9.196794746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:49.281776957 +0000 UTC m=+41.000382095" watchObservedRunningTime="2026-07-22 04:33:49.28224433 +0000 UTC m=+41.000849526" Jul 22 04:33:49.300688 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:49.299946 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" podStartSLOduration=10.66505612 podStartE2EDuration="20.299926437s" totalImagesPullingTime="9.634870317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:49.299122288 +0000 UTC m=+41.017727428" watchObservedRunningTime="2026-07-22 04:33:49.299926437 +0000 UTC m=+41.018531573" Jul 22 04:33:50.172010 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.171949 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/1.log" Jul 22 04:33:50.173110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.172940 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/0.log" Jul 22 04:33:50.173110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.172980 2572 generic.go:356] "Generic (PLEG): container finished" podID="a55ed3ab-0031-410b-bb8b-77c96710507b" containerID="2132c70899617a04ffdcc5e2c30d9676430361892bf37a9ecae44111bd6d054d" exitCode=255 Jul 22 04:33:50.173110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.173068 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" event={"ID":"a55ed3ab-0031-410b-bb8b-77c96710507b","Type":"ContainerDied","Data":"2132c70899617a04ffdcc5e2c30d9676430361892bf37a9ecae44111bd6d054d"} Jul 22 04:33:50.173357 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.173164 2572 scope.go:117] "RemoveContainer" containerID="c953e142e05f7c20b902c3336f5c408aa08ac342f75e0397f8e7a77c0855a3eb" Jul 22 04:33:50.173518 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.173489 2572 scope.go:117] "RemoveContainer" containerID="2132c70899617a04ffdcc5e2c30d9676430361892bf37a9ecae44111bd6d054d" Jul 22 04:33:50.173692 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:50.173665 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-fkwqf_openshift-console-operator(a55ed3ab-0031-410b-bb8b-77c96710507b)\"" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" podUID="a55ed3ab-0031-410b-bb8b-77c96710507b" Jul 22 04:33:50.177540 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.177515 2572 generic.go:356] "Generic (PLEG): container finished" podID="72bb9cb1-2e8a-44c3-8d61-043b7605a224" containerID="3d62213c8ceb88b423dae9ad4dc090240ab6e766132d4eb7a6786fb772d2f38d" exitCode=0 Jul 22 04:33:50.177652 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.177632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerDied","Data":"3d62213c8ceb88b423dae9ad4dc090240ab6e766132d4eb7a6786fb772d2f38d"} Jul 22 04:33:50.778647 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.777414 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r26sh"] Jul 22 04:33:50.792046 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.791956 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r26sh"] Jul 22 04:33:50.792211 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.792094 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:50.794657 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.794426 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 04:33:50.794657 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.794484 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 04:33:50.794657 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.794523 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tc9bh\"" Jul 22 04:33:50.927147 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.927110 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:50.927326 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.927294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-data-volume\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:50.927542 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.927329 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbk2\" (UniqueName: \"kubernetes.io/projected/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-api-access-7lbk2\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:50.927542 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.927364 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-crio-socket\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:50.927542 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:50.927489 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.029100 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.028799 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.029100 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.028935 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-data-volume\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.029100 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.028966 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbk2\" (UniqueName: \"kubernetes.io/projected/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-api-access-7lbk2\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.029100 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.028987 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 04:33:51.029100 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.029066 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls podName:d30365b3-2e9f-43bd-ac4c-afb30c8e228d nodeName:}" failed. No retries permitted until 2026-07-22 04:33:51.529044115 +0000 UTC m=+43.247649242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r26sh" (UID: "d30365b3-2e9f-43bd-ac4c-afb30c8e228d") : secret "insights-runtime-extractor-tls" not found Jul 22 04:33:51.029563 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.028994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-crio-socket\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.029563 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.029302 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.029563 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.029338 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-crio-socket\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.029563 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.029339 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-data-volume\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.029563 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.029446 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 04:33:51.029563 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.029505 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm podName:d30365b3-2e9f-43bd-ac4c-afb30c8e228d nodeName:}" failed. No retries permitted until 2026-07-22 04:33:51.529490716 +0000 UTC m=+43.248095829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r26sh" (UID: "d30365b3-2e9f-43bd-ac4c-afb30c8e228d") : configmap "kube-rbac-proxy" not found Jul 22 04:33:51.038901 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.038878 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbk2\" (UniqueName: \"kubernetes.io/projected/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-api-access-7lbk2\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.184595 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.183683 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/1.log" Jul 22 04:33:51.184595 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.184361 2572 scope.go:117] "RemoveContainer" containerID="2132c70899617a04ffdcc5e2c30d9676430361892bf37a9ecae44111bd6d054d" Jul 22 04:33:51.185071 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.184716 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-fkwqf_openshift-console-operator(a55ed3ab-0031-410b-bb8b-77c96710507b)\"" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" podUID="a55ed3ab-0031-410b-bb8b-77c96710507b" Jul 22 04:33:51.192024 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.191996 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" event={"ID":"72bb9cb1-2e8a-44c3-8d61-043b7605a224","Type":"ContainerStarted","Data":"91856452c95255dc384b7423c83f077be0a29eaba57e114e8007a4ea871dfdcd"} Jul 22 04:33:51.219785 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.219730 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9nnqb" podStartSLOduration=7.593291383 podStartE2EDuration="42.219709455s" totalImagesPullingTime="34.626418072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:51.218654755 +0000 UTC m=+42.937259890" watchObservedRunningTime="2026-07-22 04:33:51.219709455 +0000 UTC m=+42.938314591" Jul 22 04:33:51.535036 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.535006 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.535197 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.535177 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 04:33:51.535278 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:51.535254 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:51.535278 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.535270 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls podName:d30365b3-2e9f-43bd-ac4c-afb30c8e228d nodeName:}" failed. No retries permitted until 2026-07-22 04:33:52.535250421 +0000 UTC m=+44.253855551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r26sh" (UID: "d30365b3-2e9f-43bd-ac4c-afb30c8e228d") : secret "insights-runtime-extractor-tls" not found Jul 22 04:33:51.535416 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.535402 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 04:33:51.535472 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:51.535463 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm podName:d30365b3-2e9f-43bd-ac4c-afb30c8e228d nodeName:}" failed. No retries permitted until 2026-07-22 04:33:52.535449886 +0000 UTC m=+44.254054998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r26sh" (UID: "d30365b3-2e9f-43bd-ac4c-afb30c8e228d") : configmap "kube-rbac-proxy" not found Jul 22 04:33:52.544677 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:52.544635 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:52.545124 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:52.544702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:52.545124 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:52.544768 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 04:33:52.545124 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:52.544857 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm podName:d30365b3-2e9f-43bd-ac4c-afb30c8e228d nodeName:}" failed. No retries permitted until 2026-07-22 04:33:54.54483532 +0000 UTC m=+46.263440436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r26sh" (UID: "d30365b3-2e9f-43bd-ac4c-afb30c8e228d") : configmap "kube-rbac-proxy" not found Jul 22 04:33:52.545124 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:52.544867 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 04:33:52.545124 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:52.544919 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls podName:d30365b3-2e9f-43bd-ac4c-afb30c8e228d nodeName:}" failed. No retries permitted until 2026-07-22 04:33:54.544906203 +0000 UTC m=+46.263511318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r26sh" (UID: "d30365b3-2e9f-43bd-ac4c-afb30c8e228d") : secret "insights-runtime-extractor-tls" not found Jul 22 04:33:52.973917 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:52.973853 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k8rb5_227413da-239d-4d90-8c76-8861f42e0ac0/dns-node-resolver/0.log" Jul 22 04:33:53.773829 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:53.773802 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmptl_d00b0010-35ef-4ad0-a483-937a188385cc/node-ca/0.log" Jul 22 04:33:53.956898 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:53.956867 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:33:53.957069 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:53.957021 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 04:33:53.957069 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:53.957039 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79467c9559-6cxq7: secret "image-registry-tls" not found Jul 22 04:33:53.957157 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:53.957097 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls podName:a9058cad-5eb5-46fa-89fb-c8ccf93c5b69 nodeName:}" failed. No retries permitted until 2026-07-22 04:34:09.957081488 +0000 UTC m=+61.675686601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls") pod "image-registry-79467c9559-6cxq7" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69") : secret "image-registry-tls" not found Jul 22 04:33:54.057568 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.057484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:33:54.057707 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.057640 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 04:33:54.057744 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.057708 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls podName:4696d0cc-1616-4168-a2f8-3bb5319a2073 nodeName:}" failed. No retries permitted until 2026-07-22 04:34:10.057691015 +0000 UTC m=+61.776296129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-5bq9d" (UID: "4696d0cc-1616-4168-a2f8-3bb5319a2073") : secret "samples-operator-tls" not found Jul 22 04:33:54.158736 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.158703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:33:54.158898 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.158777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:33:54.158898 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.158804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:33:54.158898 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.158853 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 22 04:33:54.158898 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.158897 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:54.159048 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.158914 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 04:33:54.159048 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.158923 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert podName:814f7eac-85c9-4308-a905-c09798acd37c nodeName:}" failed. No retries permitted until 2026-07-22 04:34:10.158903677 +0000 UTC m=+61.877508810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-4qc7c" (UID: "814f7eac-85c9-4308-a905-c09798acd37c") : secret "networking-console-plugin-cert" not found Jul 22 04:33:54.159048 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.158974 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls podName:a19fb360-7ca0-490a-a7a6-72497334a8ba nodeName:}" failed. No retries permitted until 2026-07-22 04:34:10.158961508 +0000 UTC m=+61.877566620 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8lqlm" (UID: "a19fb360-7ca0-490a-a7a6-72497334a8ba") : secret "cluster-monitoring-operator-tls" not found Jul 22 04:33:54.159160 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.159053 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls podName:edae6ef2-897e-4f71-8df9-66e34eaaff9c nodeName:}" failed. No retries permitted until 2026-07-22 04:34:10.159045071 +0000 UTC m=+61.877650184 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls") pod "dns-default-h2wp5" (UID: "edae6ef2-897e-4f71-8df9-66e34eaaff9c") : secret "dns-default-metrics-tls" not found Jul 22 04:33:54.206864 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.206832 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrkws" event={"ID":"9c051ba5-d2d1-4add-89e0-591295ecaf5e","Type":"ContainerStarted","Data":"94f1eb01554508059b397eb280755c28e38c78646c424656e8b3c9b2de9c8ebd"} Jul 22 04:33:54.222778 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.222732 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hrkws" podStartSLOduration=40.64535629 podStartE2EDuration="45.222719368s" totalImagesPullingTime="4.577363078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:33:54.222192335 +0000 UTC m=+45.940797470" watchObservedRunningTime="2026-07-22 04:33:54.222719368 +0000 UTC m=+45.941324502" Jul 22 04:33:54.259904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.259874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:54.260040 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.259922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:33:54.260040 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.260005 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 04:33:54.260040 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.260011 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:34:10.259994912 +0000 UTC m=+61.978600029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : configmap references non-existent config key: service-ca.crt Jul 22 04:33:54.260146 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.260058 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs podName:e246d46d-6258-4941-a2c0-c71e777bf0ad nodeName:}" failed. No retries permitted until 2026-07-22 04:34:10.260045025 +0000 UTC m=+61.978650153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs") pod "router-default-dcfd7dc78-8zf2f" (UID: "e246d46d-6258-4941-a2c0-c71e777bf0ad") : secret "router-metrics-certs-default" not found Jul 22 04:33:54.562742 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.562706 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:54.562942 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.562758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:54.562942 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.562852 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 04:33:54.562942 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:54.562913 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls podName:d30365b3-2e9f-43bd-ac4c-afb30c8e228d nodeName:}" failed. No retries permitted until 2026-07-22 04:33:58.562900051 +0000 UTC m=+50.281505164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r26sh" (UID: "d30365b3-2e9f-43bd-ac4c-afb30c8e228d") : secret "insights-runtime-extractor-tls" not found Jul 22 04:33:54.563286 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.563262 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:54.774207 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:54.774179 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x5p5s_48e78c79-6a2d-44cc-8f84-c123315037b7/serve-healthcheck-canary/0.log" Jul 22 04:33:58.566807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:58.566776 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:58.567203 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:58.566833 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:33:58.567203 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:58.567168 2572 scope.go:117] "RemoveContainer" containerID="2132c70899617a04ffdcc5e2c30d9676430361892bf37a9ecae44111bd6d054d" Jul 22 04:33:58.567334 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:58.567314 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-fkwqf_openshift-console-operator(a55ed3ab-0031-410b-bb8b-77c96710507b)\"" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" podUID="a55ed3ab-0031-410b-bb8b-77c96710507b" Jul 22 04:33:58.595904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:33:58.595859 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:33:58.596073 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:58.595981 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 04:33:58.596073 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:33:58.596032 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls podName:d30365b3-2e9f-43bd-ac4c-afb30c8e228d nodeName:}" failed. No retries permitted until 2026-07-22 04:34:06.596016188 +0000 UTC m=+58.314621303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r26sh" (UID: "d30365b3-2e9f-43bd-ac4c-afb30c8e228d") : secret "insights-runtime-extractor-tls" not found Jul 22 04:34:03.095106 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:03.095071 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bpv66" Jul 22 04:34:06.663906 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:06.663870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:34:06.666106 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:06.666084 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d30365b3-2e9f-43bd-ac4c-afb30c8e228d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r26sh\" (UID: \"d30365b3-2e9f-43bd-ac4c-afb30c8e228d\") " pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:34:06.704063 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:06.704039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r26sh" Jul 22 04:34:06.850662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:06.850541 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r26sh"] Jul 22 04:34:06.853190 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:06.853167 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30365b3_2e9f_43bd_ac4c_afb30c8e228d.slice/crio-4e7cd6a4d1c194a12db6137806cc5bd9e7dea76b18e049f7991e2a70bcc47e2f WatchSource:0}: Error finding container 4e7cd6a4d1c194a12db6137806cc5bd9e7dea76b18e049f7991e2a70bcc47e2f: Status 404 returned error can't find the container with id 4e7cd6a4d1c194a12db6137806cc5bd9e7dea76b18e049f7991e2a70bcc47e2f Jul 22 04:34:07.247519 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:07.247489 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r26sh" event={"ID":"d30365b3-2e9f-43bd-ac4c-afb30c8e228d","Type":"ContainerStarted","Data":"4e7cd6a4d1c194a12db6137806cc5bd9e7dea76b18e049f7991e2a70bcc47e2f"} Jul 22 04:34:09.993515 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:09.993475 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:34:09.996316 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:09.996285 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"image-registry-79467c9559-6cxq7\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:34:10.094474 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.094428 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:34:10.097083 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.097028 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4696d0cc-1616-4168-a2f8-3bb5319a2073-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-5bq9d\" (UID: \"4696d0cc-1616-4168-a2f8-3bb5319a2073\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:34:10.195830 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.195791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:34:10.195830 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.195833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:34:10.196134 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.196109 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:34:10.198760 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.198732 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edae6ef2-897e-4f71-8df9-66e34eaaff9c-metrics-tls\") pod \"dns-default-h2wp5\" (UID: \"edae6ef2-897e-4f71-8df9-66e34eaaff9c\") " pod="openshift-dns/dns-default-h2wp5" Jul 22 04:34:10.198876 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.198738 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a19fb360-7ca0-490a-a7a6-72497334a8ba-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8lqlm\" (UID: \"a19fb360-7ca0-490a-a7a6-72497334a8ba\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:34:10.198876 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.198851 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/814f7eac-85c9-4308-a905-c09798acd37c-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-4qc7c\" (UID: \"814f7eac-85c9-4308-a905-c09798acd37c\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:34:10.249005 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.248929 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p9jfd\"" Jul 22 04:34:10.257003 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.256980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:34:10.296921 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.296886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:10.297090 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.296938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:10.297639 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.297610 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e246d46d-6258-4941-a2c0-c71e777bf0ad-service-ca-bundle\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:10.299470 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.299426 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e246d46d-6258-4941-a2c0-c71e777bf0ad-metrics-certs\") pod \"router-default-dcfd7dc78-8zf2f\" (UID: \"e246d46d-6258-4941-a2c0-c71e777bf0ad\") " pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:10.343922 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.343896 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2njs6\"" Jul 22 04:34:10.351788 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.351759 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" Jul 22 04:34:10.385121 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.385091 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dm9jd\"" Jul 22 04:34:10.393919 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.393891 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" Jul 22 04:34:10.413955 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.413931 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hhmhx\"" Jul 22 04:34:10.422746 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.422726 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h2wp5" Jul 22 04:34:10.432712 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.432694 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h7kj7\"" Jul 22 04:34:10.440585 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.440566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" Jul 22 04:34:10.519855 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.519829 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sg7t5\"" Jul 22 04:34:10.527812 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:10.527794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:12.333826 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:12.333769 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm"] Jul 22 04:34:12.352879 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:12.352778 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79467c9559-6cxq7"] Jul 22 04:34:12.480164 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:12.480076 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19fb360_7ca0_490a_a7a6_72497334a8ba.slice/crio-6ff9103e7aa5c5b0671ee1e348dcb4c4543b1c55abad9e6e871523f49b782b45 WatchSource:0}: Error finding container 6ff9103e7aa5c5b0671ee1e348dcb4c4543b1c55abad9e6e871523f49b782b45: Status 404 returned error can't find the container with id 6ff9103e7aa5c5b0671ee1e348dcb4c4543b1c55abad9e6e871523f49b782b45 Jul 22 04:34:12.480824 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:12.480800 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9058cad_5eb5_46fa_89fb_c8ccf93c5b69.slice/crio-70d8e052139ce360dd4ea055fcfe13ae08de9b92ee8e1b8999537b11c466c84f WatchSource:0}: Error finding container 70d8e052139ce360dd4ea055fcfe13ae08de9b92ee8e1b8999537b11c466c84f: Status 404 returned error can't find the container with id 70d8e052139ce360dd4ea055fcfe13ae08de9b92ee8e1b8999537b11c466c84f Jul 22 04:34:12.576669 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:12.576637 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d"] Jul 22 04:34:12.581092 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:12.580999 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-dcfd7dc78-8zf2f"] Jul 22 04:34:12.583487 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:12.583457 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode246d46d_6258_4941_a2c0_c71e777bf0ad.slice/crio-bdbcbe2e250500f1908f9a79cb63b7b1bcd343a870e4d44906a21df7de545677 WatchSource:0}: Error finding container bdbcbe2e250500f1908f9a79cb63b7b1bcd343a870e4d44906a21df7de545677: Status 404 returned error can't find the container with id bdbcbe2e250500f1908f9a79cb63b7b1bcd343a870e4d44906a21df7de545677 Jul 22 04:34:12.587529 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:12.587511 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c"] Jul 22 04:34:12.588817 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:12.588795 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h2wp5"] Jul 22 04:34:12.592753 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:12.592726 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedae6ef2_897e_4f71_8df9_66e34eaaff9c.slice/crio-29e7ca199f0c4a51b7817db1eff2a3e837d07408dd4ab20062b2e713e2ad91ce WatchSource:0}: Error finding container 29e7ca199f0c4a51b7817db1eff2a3e837d07408dd4ab20062b2e713e2ad91ce: Status 404 returned error can't find the container with id 29e7ca199f0c4a51b7817db1eff2a3e837d07408dd4ab20062b2e713e2ad91ce Jul 22 04:34:12.594471 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:12.594422 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814f7eac_85c9_4308_a905_c09798acd37c.slice/crio-403fda86c36228a57729e0b7e24112ae246cb7daa1ddb9135a344fa1c90ee845 WatchSource:0}: Error finding container 403fda86c36228a57729e0b7e24112ae246cb7daa1ddb9135a344fa1c90ee845: Status 404 returned error can't find the container with id 403fda86c36228a57729e0b7e24112ae246cb7daa1ddb9135a344fa1c90ee845 Jul 22 04:34:12.795535 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:12.795507 2572 scope.go:117] "RemoveContainer" containerID="2132c70899617a04ffdcc5e2c30d9676430361892bf37a9ecae44111bd6d054d" Jul 22 04:34:13.277415 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.277359 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" event={"ID":"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69","Type":"ContainerStarted","Data":"1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2"} Jul 22 04:34:13.277582 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.277442 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" event={"ID":"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69","Type":"ContainerStarted","Data":"70d8e052139ce360dd4ea055fcfe13ae08de9b92ee8e1b8999537b11c466c84f"} Jul 22 04:34:13.277688 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.277644 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:34:13.280113 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.280023 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r26sh" event={"ID":"d30365b3-2e9f-43bd-ac4c-afb30c8e228d","Type":"ContainerStarted","Data":"5ba02ff6bb4394b8bfb1d05f0a9776dbd05b15929ebe458931183afc6289c33c"} Jul 22 04:34:13.282387 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.282303 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" event={"ID":"4696d0cc-1616-4168-a2f8-3bb5319a2073","Type":"ContainerStarted","Data":"958641a46a7a24610b863641b7f5f93ff0e8d331a4a7fd97697d391c573ea911"} Jul 22 04:34:13.283720 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.283675 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" event={"ID":"a19fb360-7ca0-490a-a7a6-72497334a8ba","Type":"ContainerStarted","Data":"6ff9103e7aa5c5b0671ee1e348dcb4c4543b1c55abad9e6e871523f49b782b45"} Jul 22 04:34:13.286703 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.286677 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" event={"ID":"e246d46d-6258-4941-a2c0-c71e777bf0ad","Type":"ContainerStarted","Data":"111fbc9792bb32a14eb0e019a0b7df2f3b0ad8e297b6a148cda1bbcef8d4349a"} Jul 22 04:34:13.286807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.286720 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" event={"ID":"e246d46d-6258-4941-a2c0-c71e777bf0ad","Type":"ContainerStarted","Data":"bdbcbe2e250500f1908f9a79cb63b7b1bcd343a870e4d44906a21df7de545677"} Jul 22 04:34:13.289144 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.289062 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:34:13.289732 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.289714 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/1.log" Jul 22 04:34:13.289818 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.289746 2572 generic.go:356] "Generic (PLEG): container finished" podID="a55ed3ab-0031-410b-bb8b-77c96710507b" containerID="26d9b9e466bc2fdf18ea5285a711591685f2c9b3cb87aca47af6a1040a1349a4" exitCode=255 Jul 22 04:34:13.289881 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.289815 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" event={"ID":"a55ed3ab-0031-410b-bb8b-77c96710507b","Type":"ContainerDied","Data":"26d9b9e466bc2fdf18ea5285a711591685f2c9b3cb87aca47af6a1040a1349a4"} Jul 22 04:34:13.289881 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.289855 2572 scope.go:117] "RemoveContainer" containerID="2132c70899617a04ffdcc5e2c30d9676430361892bf37a9ecae44111bd6d054d" Jul 22 04:34:13.290241 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.290224 2572 scope.go:117] "RemoveContainer" containerID="26d9b9e466bc2fdf18ea5285a711591685f2c9b3cb87aca47af6a1040a1349a4" Jul 22 04:34:13.290434 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:34:13.290412 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-fkwqf_openshift-console-operator(a55ed3ab-0031-410b-bb8b-77c96710507b)\"" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" podUID="a55ed3ab-0031-410b-bb8b-77c96710507b" Jul 22 04:34:13.292727 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.292671 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h2wp5" event={"ID":"edae6ef2-897e-4f71-8df9-66e34eaaff9c","Type":"ContainerStarted","Data":"29e7ca199f0c4a51b7817db1eff2a3e837d07408dd4ab20062b2e713e2ad91ce"} Jul 22 04:34:13.294251 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.294227 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" event={"ID":"814f7eac-85c9-4308-a905-c09798acd37c","Type":"ContainerStarted","Data":"403fda86c36228a57729e0b7e24112ae246cb7daa1ddb9135a344fa1c90ee845"} Jul 22 04:34:13.297692 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.297436 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" podStartSLOduration=67.297420025 podStartE2EDuration="1m7.297420025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:13.296029331 +0000 UTC m=+65.014634485" watchObservedRunningTime="2026-07-22 04:34:13.297420025 +0000 UTC m=+65.016025159" Jul 22 04:34:13.330157 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.329614 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" podStartSLOduration=44.329593567 podStartE2EDuration="44.329593567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:13.313588126 +0000 UTC m=+65.032193263" watchObservedRunningTime="2026-07-22 04:34:13.329593567 +0000 UTC m=+65.048198703" Jul 22 04:34:13.528808 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.528719 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:13.531817 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.531595 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:13.627558 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.627515 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:34:13.635892 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.635832 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c2de1e1-0496-4eda-b5e5-12b1c181582e-metrics-certs\") pod \"network-metrics-daemon-wsxd4\" (UID: \"0c2de1e1-0496-4eda-b5e5-12b1c181582e\") " pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:34:13.920117 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.920042 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ljtjg\"" Jul 22 04:34:13.928309 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:13.928085 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wsxd4" Jul 22 04:34:14.301874 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:14.301842 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:34:14.303392 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:14.303341 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:14.308058 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:14.304716 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-dcfd7dc78-8zf2f" Jul 22 04:34:18.232262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:18.232231 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wsxd4"] Jul 22 04:34:18.566551 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:18.566514 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:34:18.566753 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:18.566583 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:34:18.567000 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:18.566975 2572 scope.go:117] "RemoveContainer" containerID="26d9b9e466bc2fdf18ea5285a711591685f2c9b3cb87aca47af6a1040a1349a4" Jul 22 04:34:18.567198 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:34:18.567172 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-fkwqf_openshift-console-operator(a55ed3ab-0031-410b-bb8b-77c96710507b)\"" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" podUID="a55ed3ab-0031-410b-bb8b-77c96710507b" Jul 22 04:34:18.667457 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:18.667427 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2de1e1_0496_4eda_b5e5_12b1c181582e.slice/crio-8500ba191806c09ad1a74d692294f3d8fc0b32793584aaa380117bdb812b4de8 WatchSource:0}: Error finding container 8500ba191806c09ad1a74d692294f3d8fc0b32793584aaa380117bdb812b4de8: Status 404 returned error can't find the container with id 8500ba191806c09ad1a74d692294f3d8fc0b32793584aaa380117bdb812b4de8 Jul 22 04:34:19.325610 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.325270 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" event={"ID":"4696d0cc-1616-4168-a2f8-3bb5319a2073","Type":"ContainerStarted","Data":"0dbabdcbb5d703263ad74974dff066e6160ae15b400f8e958cbb7bd93b97c221"} Jul 22 04:34:19.325610 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.325348 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" event={"ID":"4696d0cc-1616-4168-a2f8-3bb5319a2073","Type":"ContainerStarted","Data":"43cc44f64d40ee241a0975a4961654a85f8dc677f833e8be28410874ce90f584"} Jul 22 04:34:19.327423 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.327392 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" event={"ID":"a19fb360-7ca0-490a-a7a6-72497334a8ba","Type":"ContainerStarted","Data":"f28fdfbcfdffc54ecf7a380dbf5e179d5bbe8036ec348caca3cf63e1a3882364"} Jul 22 04:34:19.328736 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.328693 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wsxd4" event={"ID":"0c2de1e1-0496-4eda-b5e5-12b1c181582e","Type":"ContainerStarted","Data":"8500ba191806c09ad1a74d692294f3d8fc0b32793584aaa380117bdb812b4de8"} Jul 22 04:34:19.331511 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.331480 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h2wp5" event={"ID":"edae6ef2-897e-4f71-8df9-66e34eaaff9c","Type":"ContainerStarted","Data":"5de0c4fdb1a9362d54e0fee92e43b7627b8075c363e55b656d045770878e3484"} Jul 22 04:34:19.331609 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.331529 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h2wp5" event={"ID":"edae6ef2-897e-4f71-8df9-66e34eaaff9c","Type":"ContainerStarted","Data":"8d5f34bd20d5bc3274f0780c745694ccd535dfea53e8ab582b5050d7a0ae81db"} Jul 22 04:34:19.331609 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.331561 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h2wp5" Jul 22 04:34:19.333591 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.333566 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" event={"ID":"814f7eac-85c9-4308-a905-c09798acd37c","Type":"ContainerStarted","Data":"1a99a25f2f34a52253bbbe266a3459ae3f78dcf7d63af09325435995b3b549a7"} Jul 22 04:34:19.336086 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.336061 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r26sh" event={"ID":"d30365b3-2e9f-43bd-ac4c-afb30c8e228d","Type":"ContainerStarted","Data":"322fac63b463110165d1db7b5dd855f7e9020d22cdbd6f4fcdac6b95963e21f6"} Jul 22 04:34:19.341533 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.341473 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-5bq9d" podStartSLOduration=44.305328625 podStartE2EDuration="50.341455819s" totalImagesPullingTime="6.036127194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:19.340974422 +0000 UTC m=+71.059579559" watchObservedRunningTime="2026-07-22 04:34:19.341455819 +0000 UTC m=+71.060060956" Jul 22 04:34:19.355724 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.355653 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-4qc7c" podStartSLOduration=44.885041231 podStartE2EDuration="50.355635126s" totalImagesPullingTime="5.470593895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:19.355036819 +0000 UTC m=+71.073641955" watchObservedRunningTime="2026-07-22 04:34:19.355635126 +0000 UTC m=+71.074240265" Jul 22 04:34:19.373268 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.373211 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8lqlm" podStartSLOduration=44.215901702 podStartE2EDuration="50.373188727s" totalImagesPullingTime="6.157287025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:19.371675002 +0000 UTC m=+71.090280139" watchObservedRunningTime="2026-07-22 04:34:19.373188727 +0000 UTC m=+71.091793863" Jul 22 04:34:19.396540 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:19.396437 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-h2wp5" podStartSLOduration=35.32560988 podStartE2EDuration="41.396416391s" totalImagesPullingTime="6.070806511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:19.394847214 +0000 UTC m=+71.113452349" watchObservedRunningTime="2026-07-22 04:34:19.396416391 +0000 UTC m=+71.115021528" Jul 22 04:34:20.181725 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:20.181640 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-scr4t" Jul 22 04:34:20.342597 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:20.342561 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wsxd4" event={"ID":"0c2de1e1-0496-4eda-b5e5-12b1c181582e","Type":"ContainerStarted","Data":"99e6f3386b9cf0f42c44369ee82bdd927cb27cb94f4cf082bf03cf271aebe461"} Jul 22 04:34:20.343070 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:20.342626 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wsxd4" event={"ID":"0c2de1e1-0496-4eda-b5e5-12b1c181582e","Type":"ContainerStarted","Data":"a521d0ad2a7e2623e2c653ec8a3135019dcf19d8c67e8e2733a88b2d358df50b"} Jul 22 04:34:20.357757 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:20.357556 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wsxd4" podStartSLOduration=70.256473848 podStartE2EDuration="1m11.357536988s" totalImagesPullingTime="1.10106314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:20.357136112 +0000 UTC m=+72.075741248" watchObservedRunningTime="2026-07-22 04:34:20.357536988 +0000 UTC m=+72.076142124" Jul 22 04:34:23.356498 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:23.356456 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r26sh" event={"ID":"d30365b3-2e9f-43bd-ac4c-afb30c8e228d","Type":"ContainerStarted","Data":"233dadcc8cda7a5fd647ad95cfe7f025ba6a21b11108fce705618581fdba50b3"} Jul 22 04:34:23.374633 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:23.374581 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r26sh" podStartSLOduration=18.065320293 podStartE2EDuration="33.374567238s" totalImagesPullingTime="15.309246945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 04:33:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:23.372767362 +0000 UTC m=+75.091372496" watchObservedRunningTime="2026-07-22 04:34:23.374567238 +0000 UTC m=+75.093172377" Jul 22 04:34:26.342211 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.342175 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4"] Jul 22 04:34:26.344919 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.344901 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.347287 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.347267 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 04:34:26.348187 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.348168 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 04:34:26.348262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.348173 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-q6dg4\"" Jul 22 04:34:26.348299 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.348257 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 04:34:26.355731 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.355708 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4"] Jul 22 04:34:26.437020 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.436987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa088c82-ac38-444a-ac7e-23ac88cb6759-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.437173 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.437039 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqf8l\" (UniqueName: \"kubernetes.io/projected/fa088c82-ac38-444a-ac7e-23ac88cb6759-kube-api-access-zqf8l\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.437173 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.437064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa088c82-ac38-444a-ac7e-23ac88cb6759-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.437173 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.437146 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa088c82-ac38-444a-ac7e-23ac88cb6759-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.538089 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.538057 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa088c82-ac38-444a-ac7e-23ac88cb6759-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.538220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.538103 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa088c82-ac38-444a-ac7e-23ac88cb6759-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.538220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.538137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa088c82-ac38-444a-ac7e-23ac88cb6759-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.538220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.538196 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqf8l\" (UniqueName: \"kubernetes.io/projected/fa088c82-ac38-444a-ac7e-23ac88cb6759-kube-api-access-zqf8l\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.538842 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.538815 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa088c82-ac38-444a-ac7e-23ac88cb6759-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.540651 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.540623 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa088c82-ac38-444a-ac7e-23ac88cb6759-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.540726 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.540673 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa088c82-ac38-444a-ac7e-23ac88cb6759-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.548664 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.548641 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqf8l\" (UniqueName: \"kubernetes.io/projected/fa088c82-ac38-444a-ac7e-23ac88cb6759-kube-api-access-zqf8l\") pod \"prometheus-operator-5945b5cfd-nmxw4\" (UID: \"fa088c82-ac38-444a-ac7e-23ac88cb6759\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.654011 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.653933 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" Jul 22 04:34:26.782183 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:26.782150 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4"] Jul 22 04:34:26.785496 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:26.785465 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa088c82_ac38_444a_ac7e_23ac88cb6759.slice/crio-43cb3e541e886083cdc503bf18c21da95bd1641a06a6a079f6e152a361793b23 WatchSource:0}: Error finding container 43cb3e541e886083cdc503bf18c21da95bd1641a06a6a079f6e152a361793b23: Status 404 returned error can't find the container with id 43cb3e541e886083cdc503bf18c21da95bd1641a06a6a079f6e152a361793b23 Jul 22 04:34:27.372589 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:27.372553 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" event={"ID":"fa088c82-ac38-444a-ac7e-23ac88cb6759","Type":"ContainerStarted","Data":"43cb3e541e886083cdc503bf18c21da95bd1641a06a6a079f6e152a361793b23"} Jul 22 04:34:28.378188 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:28.378097 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" event={"ID":"fa088c82-ac38-444a-ac7e-23ac88cb6759","Type":"ContainerStarted","Data":"b444153e762ae49ff7b816f61d1cd7d8a143897a92253506e75145c6e61cd9de"} Jul 22 04:34:28.378188 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:28.378153 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" event={"ID":"fa088c82-ac38-444a-ac7e-23ac88cb6759","Type":"ContainerStarted","Data":"256b0b93b03dbf0a7a9dbfcee2142021e8f3884b55fe041b87a0117276e5307a"} Jul 22 04:34:28.394553 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:28.394508 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-nmxw4" podStartSLOduration=1.139074136 podStartE2EDuration="2.39449212s" totalImagesPullingTime="1.255417984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:28.392700156 +0000 UTC m=+80.111305290" watchObservedRunningTime="2026-07-22 04:34:28.39449212 +0000 UTC m=+80.113097254" Jul 22 04:34:29.344644 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:29.344613 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h2wp5" Jul 22 04:34:29.795241 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:29.795211 2572 scope.go:117] "RemoveContainer" containerID="26d9b9e466bc2fdf18ea5285a711591685f2c9b3cb87aca47af6a1040a1349a4" Jul 22 04:34:29.795622 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:34:29.795369 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-565b7b569d-fkwqf_openshift-console-operator(a55ed3ab-0031-410b-bb8b-77c96710507b)\"" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" podUID="a55ed3ab-0031-410b-bb8b-77c96710507b" Jul 22 04:34:30.708178 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.708066 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm"] Jul 22 04:34:30.711248 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.711231 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.713489 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.713466 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 04:34:30.713596 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.713538 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 04:34:30.713596 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.713583 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h2gsl\"" Jul 22 04:34:30.722873 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.722853 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm"] Jul 22 04:34:30.744828 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.744807 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-92jgj"] Jul 22 04:34:30.747837 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.747822 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.750018 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.749996 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zm9xl\"" Jul 22 04:34:30.750120 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.750040 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 04:34:30.750120 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.750054 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 04:34:30.750265 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.750250 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 04:34:30.872966 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.872936 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-root\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.872977 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873013 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873034 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgtt4\" (UniqueName: \"kubernetes.io/projected/e3847ee0-c179-4e80-a233-f9c66722e203-kube-api-access-sgtt4\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873114 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rx2r\" (UniqueName: \"kubernetes.io/projected/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-kube-api-access-2rx2r\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873163 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-wtmp\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873192 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873220 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-tls\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873265 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-sys\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873287 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-accelerators-collector-config\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873328 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-textfile\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.873417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.873354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3847ee0-c179-4e80-a233-f9c66722e203-metrics-client-ca\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.973899 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.973831 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.973899 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.973873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.973899 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.973892 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgtt4\" (UniqueName: \"kubernetes.io/projected/e3847ee0-c179-4e80-a233-f9c66722e203-kube-api-access-sgtt4\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974121 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.973922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.974121 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.973947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rx2r\" (UniqueName: \"kubernetes.io/projected/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-kube-api-access-2rx2r\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.974121 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.973967 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-wtmp\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974121 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.973996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.974121 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974022 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-tls\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974367 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974147 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-wtmp\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974367 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:34:30.974245 2572 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 22 04:34:30.974367 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:34:30.974308 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-tls podName:ca60eca3-d74f-46bb-82eb-e6d4d7fd2781 nodeName:}" failed. No retries permitted until 2026-07-22 04:34:31.474286446 +0000 UTC m=+83.192891576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-h8tfm" (UID: "ca60eca3-d74f-46bb-82eb-e6d4d7fd2781") : secret "openshift-state-metrics-tls" not found Jul 22 04:34:30.974561 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974366 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-sys\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974561 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-accelerators-collector-config\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974561 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974442 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-sys\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974561 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-textfile\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974561 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3847ee0-c179-4e80-a233-f9c66722e203-metrics-client-ca\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974561 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-root\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974845 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974631 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3847ee0-c179-4e80-a233-f9c66722e203-root\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.974845 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974784 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.974845 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.974821 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-textfile\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.975024 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.975003 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-accelerators-collector-config\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.975190 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.975165 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3847ee0-c179-4e80-a233-f9c66722e203-metrics-client-ca\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.976296 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.976273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.976436 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.976418 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3847ee0-c179-4e80-a233-f9c66722e203-node-exporter-tls\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.976572 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.976554 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:30.981852 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.981830 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgtt4\" (UniqueName: \"kubernetes.io/projected/e3847ee0-c179-4e80-a233-f9c66722e203-kube-api-access-sgtt4\") pod \"node-exporter-92jgj\" (UID: \"e3847ee0-c179-4e80-a233-f9c66722e203\") " pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:30.982093 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:30.982067 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rx2r\" (UniqueName: \"kubernetes.io/projected/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-kube-api-access-2rx2r\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:31.055642 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.055605 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-92jgj" Jul 22 04:34:31.066915 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:31.066887 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3847ee0_c179_4e80_a233_f9c66722e203.slice/crio-ac5aa2025060e0c2b3a367def95a3831f0427c6ab4cd5017f99b9c6860a1b5c1 WatchSource:0}: Error finding container ac5aa2025060e0c2b3a367def95a3831f0427c6ab4cd5017f99b9c6860a1b5c1: Status 404 returned error can't find the container with id ac5aa2025060e0c2b3a367def95a3831f0427c6ab4cd5017f99b9c6860a1b5c1 Jul 22 04:34:31.391051 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.391015 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-92jgj" event={"ID":"e3847ee0-c179-4e80-a233-f9c66722e203","Type":"ContainerStarted","Data":"ac5aa2025060e0c2b3a367def95a3831f0427c6ab4cd5017f99b9c6860a1b5c1"} Jul 22 04:34:31.480335 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.480298 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:31.482748 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.482723 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca60eca3-d74f-46bb-82eb-e6d4d7fd2781-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-h8tfm\" (UID: \"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:31.620851 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.620809 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" Jul 22 04:34:31.771090 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.771001 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:34:31.777758 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.777726 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.780294 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.780269 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 04:34:31.780597 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.780576 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 04:34:31.780705 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.780599 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 04:34:31.780705 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.780603 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l5qtd\"" Jul 22 04:34:31.780705 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.780625 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 04:34:31.780897 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.780877 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 04:34:31.781012 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.780919 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 04:34:31.781012 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.780986 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 04:34:31.781125 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.781050 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 04:34:31.781125 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.781110 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 04:34:31.785655 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.785477 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm"] Jul 22 04:34:31.789729 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.789650 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.884948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.884992 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885053 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885106 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-web-config\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885209 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885238 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885267 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885289 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnmll\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-kube-api-access-dnmll\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885314 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.885524 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.885342 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-out\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.986048 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.985974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.986197 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.986055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-web-config\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.986197 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.986078 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.986197 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.986099 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.986197 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.986117 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.986197 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.986145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.988531 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.988493 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnmll\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-kube-api-access-dnmll\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.988638 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.988553 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.988638 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.988590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-out\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.988762 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.988639 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.988762 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.988669 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.988762 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.988699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.988762 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.988723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.989730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.989511 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.989833 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.989809 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.989949 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:34:31.989931 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle podName:b1a29e76-e036-4a08-aeb3-29dfe40bf0b2 nodeName:}" failed. No retries permitted until 2026-07-22 04:34:32.489909474 +0000 UTC m=+84.208514605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2") : configmap references non-existent config key: ca-bundle.crt Jul 22 04:34:31.990812 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.990785 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.991957 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.991930 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-out\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.992304 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.992272 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-web-config\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.992834 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.992813 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.993218 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.993200 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.993295 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.993212 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.993669 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.993646 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.994201 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.994144 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.995191 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.995168 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:31.997994 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:31.997970 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnmll\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-kube-api-access-dnmll\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:32.397059 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.397023 2572 generic.go:356] "Generic (PLEG): container finished" podID="e3847ee0-c179-4e80-a233-f9c66722e203" containerID="33e7dcd504f1a400ea4df8351f716472e43769f16be3a81b7537d4943fec8ba7" exitCode=0 Jul 22 04:34:32.397203 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.397098 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-92jgj" event={"ID":"e3847ee0-c179-4e80-a233-f9c66722e203","Type":"ContainerDied","Data":"33e7dcd504f1a400ea4df8351f716472e43769f16be3a81b7537d4943fec8ba7"} Jul 22 04:34:32.399072 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.399044 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" event={"ID":"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781","Type":"ContainerStarted","Data":"7cfdcd4a3444cb1afc7c9628cf2b0f66be6c9a7927d8a8fe705c6079cfcfaf2d"} Jul 22 04:34:32.399178 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.399101 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" event={"ID":"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781","Type":"ContainerStarted","Data":"4f52db60dbcf9765ea79722bc57a05bfd2aec816eed4af6457d2ba0821487cc0"} Jul 22 04:34:32.399178 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.399132 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" event={"ID":"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781","Type":"ContainerStarted","Data":"35f7bf48e43a66785a5ad924551ba63c8eaaef649022a768ea40702eec147e29"} Jul 22 04:34:32.494339 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.494307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:32.495091 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.495071 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:32.694401 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.694304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:34:32.840964 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:32.840927 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:34:32.844448 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:34:32.844416 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a29e76_e036_4a08_aeb3_29dfe40bf0b2.slice/crio-5195854db7355cd71dc0c2d720ea9ce10394902b1bd6c3dabec91a933ad4b5cb WatchSource:0}: Error finding container 5195854db7355cd71dc0c2d720ea9ce10394902b1bd6c3dabec91a933ad4b5cb: Status 404 returned error can't find the container with id 5195854db7355cd71dc0c2d720ea9ce10394902b1bd6c3dabec91a933ad4b5cb Jul 22 04:34:33.405960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:33.405924 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-92jgj" event={"ID":"e3847ee0-c179-4e80-a233-f9c66722e203","Type":"ContainerStarted","Data":"e0993c3a04c1acb6d7e756f940e0a4a1ca152cb56c76d305ae0615b194c3e849"} Jul 22 04:34:33.406244 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:33.406006 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-92jgj" event={"ID":"e3847ee0-c179-4e80-a233-f9c66722e203","Type":"ContainerStarted","Data":"ec5ef2dcc7215fb2e9f56c2415fe802814e9a2ef114d83dd61d9a969c21f4dcb"} Jul 22 04:34:33.407270 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:33.407245 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerStarted","Data":"5195854db7355cd71dc0c2d720ea9ce10394902b1bd6c3dabec91a933ad4b5cb"} Jul 22 04:34:33.425082 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:33.425044 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-92jgj" podStartSLOduration=2.32643286 podStartE2EDuration="3.425029294s" totalImagesPullingTime="1.036080621s" totalInitContainerRuntime="62.515813ms" isStatefulPod=true podCreationTimestamp="2026-07-22 04:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:33.424156931 +0000 UTC m=+85.142762069" watchObservedRunningTime="2026-07-22 04:34:33.425029294 +0000 UTC m=+85.143634462" Jul 22 04:34:34.308489 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:34.308457 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:34:34.414685 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:34.414650 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" event={"ID":"ca60eca3-d74f-46bb-82eb-e6d4d7fd2781","Type":"ContainerStarted","Data":"3c0f1c13b8fd4e195d2d26b88cca23619522e72391498952fa1be88f2a9c3f01"} Jul 22 04:34:34.430192 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:34.430150 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-h8tfm" podStartSLOduration=3.140658678 podStartE2EDuration="4.430134762s" totalImagesPullingTime="1.289476084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:34.429823342 +0000 UTC m=+86.148428492" watchObservedRunningTime="2026-07-22 04:34:34.430134762 +0000 UTC m=+86.148739907" Jul 22 04:34:35.421061 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:35.421027 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerID="7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e" exitCode=0 Jul 22 04:34:35.421435 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:35.421164 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerDied","Data":"7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e"} Jul 22 04:34:37.435365 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:37.435334 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerStarted","Data":"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9"} Jul 22 04:34:37.435682 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:37.435417 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerStarted","Data":"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37"} Jul 22 04:34:37.435682 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:37.435477 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerStarted","Data":"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2"} Jul 22 04:34:37.435682 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:37.435503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerStarted","Data":"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e"} Jul 22 04:34:37.435682 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:37.435527 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerStarted","Data":"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032"} Jul 22 04:34:38.442849 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:38.442760 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerStarted","Data":"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d"} Jul 22 04:34:38.470803 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:38.470247 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.5054418529999998 podStartE2EDuration="7.470226621s" totalImagesPullingTime="3.95030291s" totalInitContainerRuntime="14.481858ms" isStatefulPod=false podCreationTimestamp="2026-07-22 04:34:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:38.468091181 +0000 UTC m=+90.186696316" watchObservedRunningTime="2026-07-22 04:34:38.470226621 +0000 UTC m=+90.188831756" Jul 22 04:34:40.756135 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:40.756092 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79467c9559-6cxq7"] Jul 22 04:34:44.795722 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:44.795691 2572 scope.go:117] "RemoveContainer" containerID="26d9b9e466bc2fdf18ea5285a711591685f2c9b3cb87aca47af6a1040a1349a4" Jul 22 04:34:45.471665 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:45.471635 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:34:45.471840 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:45.471745 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" event={"ID":"a55ed3ab-0031-410b-bb8b-77c96710507b","Type":"ContainerStarted","Data":"abd49bc1413b18e62d07e6b8b274627066f0dcdbbf1b8371b1976f7aa93fc748"} Jul 22 04:34:45.472081 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:45.472059 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:34:45.488168 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:45.488122 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" podStartSLOduration=66.768164187 podStartE2EDuration="1m16.488106671s" totalImagesPullingTime="9.719942484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:34:45.487016417 +0000 UTC m=+97.205621560" watchObservedRunningTime="2026-07-22 04:34:45.488106671 +0000 UTC m=+97.206711804" Jul 22 04:34:45.825575 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:34:45.825538 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-fkwqf" Jul 22 04:35:00.531000 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:00.530969 2572 generic.go:356] "Generic (PLEG): container finished" podID="d5c5da81-0039-4b4e-83c4-3f701156340f" containerID="65632ca85e5d23562515718572f8f6522d68822101f62401c6bd9e12a3760a11" exitCode=0 Jul 22 04:35:00.531361 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:00.531019 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" event={"ID":"d5c5da81-0039-4b4e-83c4-3f701156340f","Type":"ContainerDied","Data":"65632ca85e5d23562515718572f8f6522d68822101f62401c6bd9e12a3760a11"} Jul 22 04:35:00.531361 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:00.531292 2572 scope.go:117] "RemoveContainer" containerID="65632ca85e5d23562515718572f8f6522d68822101f62401c6bd9e12a3760a11" Jul 22 04:35:01.536694 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:01.536656 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-sdlct" event={"ID":"d5c5da81-0039-4b4e-83c4-3f701156340f","Type":"ContainerStarted","Data":"32f3447212506033253fe3823b44cc59d1b228a4e9f4267e66c52c642309fbb5"} Jul 22 04:35:05.776457 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:05.776416 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" podUID="a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" containerName="registry" containerID="cri-o://1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2" gracePeriod=30 Jul 22 04:35:06.019944 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.019919 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:35:06.065708 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.065642 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-ca-trust-extracted\") pod \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " Jul 22 04:35:06.065708 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.065681 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-installation-pull-secrets\") pod \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " Jul 22 04:35:06.065708 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.065707 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-image-registry-private-configuration\") pod \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " Jul 22 04:35:06.065954 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.065729 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58lhc\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-kube-api-access-58lhc\") pod \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " Jul 22 04:35:06.065954 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.065764 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-bound-sa-token\") pod \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " Jul 22 04:35:06.065954 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.065800 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-trusted-ca\") pod \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " Jul 22 04:35:06.065954 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.065840 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-certificates\") pod \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " Jul 22 04:35:06.065954 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.065931 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") pod \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\" (UID: \"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69\") " Jul 22 04:35:06.066279 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.066254 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:35:06.066544 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.066508 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:35:06.068356 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.068307 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:06.068755 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.068721 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:06.068755 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.068724 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-kube-api-access-58lhc" (OuterVolumeSpecName: "kube-api-access-58lhc") pod "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69"). InnerVolumeSpecName "kube-api-access-58lhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:35:06.068901 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.068881 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:35:06.068901 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.068887 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:35:06.074720 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.074696 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" (UID: "a9058cad-5eb5-46fa-89fb-c8ccf93c5b69"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:35:06.166720 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.166684 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-installation-pull-secrets\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:06.166720 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.166717 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-image-registry-private-configuration\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:06.166904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.166731 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lhc\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-kube-api-access-58lhc\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:06.166904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.166744 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-bound-sa-token\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:06.166904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.166755 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-trusted-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:06.166904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.166766 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-certificates\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:06.166904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.166777 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-registry-tls\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:06.166904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.166790 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69-ca-trust-extracted\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:06.558465 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.558430 2572 generic.go:356] "Generic (PLEG): container finished" podID="a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" containerID="1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2" exitCode=0 Jul 22 04:35:06.558667 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.558476 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" Jul 22 04:35:06.558667 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.558489 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" event={"ID":"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69","Type":"ContainerDied","Data":"1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2"} Jul 22 04:35:06.558667 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.558540 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79467c9559-6cxq7" event={"ID":"a9058cad-5eb5-46fa-89fb-c8ccf93c5b69","Type":"ContainerDied","Data":"70d8e052139ce360dd4ea055fcfe13ae08de9b92ee8e1b8999537b11c466c84f"} Jul 22 04:35:06.558667 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.558565 2572 scope.go:117] "RemoveContainer" containerID="1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2" Jul 22 04:35:06.567302 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.567287 2572 scope.go:117] "RemoveContainer" containerID="1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2" Jul 22 04:35:06.567544 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:35:06.567527 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2\": container with ID starting with 1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2 not found: ID does not exist" containerID="1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2" Jul 22 04:35:06.567593 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.567553 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2"} err="failed to get container status \"1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2\": rpc error: code = NotFound desc = could not find container \"1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2\": container with ID starting with 1470c82f5e2f4e1b313d952b607d38a39b5f66d94ed1450f7e5109cb1ea5f0b2 not found: ID does not exist" Jul 22 04:35:06.578939 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.578914 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79467c9559-6cxq7"] Jul 22 04:35:06.582167 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.582146 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79467c9559-6cxq7"] Jul 22 04:35:06.799573 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:06.799535 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" path="/var/lib/kubelet/pods/a9058cad-5eb5-46fa-89fb-c8ccf93c5b69/volumes" Jul 22 04:35:07.099975 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.099940 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6499575489-ln7cf"] Jul 22 04:35:07.100261 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.100248 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" containerName="registry" Jul 22 04:35:07.100303 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.100263 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" containerName="registry" Jul 22 04:35:07.100336 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.100321 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9058cad-5eb5-46fa-89fb-c8ccf93c5b69" containerName="registry" Jul 22 04:35:07.105679 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.105654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.108225 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.108202 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 04:35:07.108526 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.108507 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 04:35:07.108609 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.108546 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 04:35:07.109501 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.109450 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vn4zg\"" Jul 22 04:35:07.109501 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.109471 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 04:35:07.109685 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.109470 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 04:35:07.109685 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.109526 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 04:35:07.109769 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.109747 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 04:35:07.114237 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.114215 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6499575489-ln7cf"] Jul 22 04:35:07.114423 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.114403 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 04:35:07.176408 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.176358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-service-ca\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.176533 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.176431 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgmr\" (UniqueName: \"kubernetes.io/projected/686dc483-33bd-4414-a2a2-89c0875785b7-kube-api-access-gkgmr\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.176533 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.176480 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-oauth-serving-cert\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.176533 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.176516 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-serving-cert\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.176656 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.176540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-oauth-config\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.176656 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.176624 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-trusted-ca-bundle\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.176731 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.176652 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-console-config\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.277412 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.277358 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-service-ca\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.277577 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.277425 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgmr\" (UniqueName: \"kubernetes.io/projected/686dc483-33bd-4414-a2a2-89c0875785b7-kube-api-access-gkgmr\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.277577 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.277444 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-oauth-serving-cert\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.277577 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.277462 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-serving-cert\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.277577 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.277480 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-oauth-config\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.277577 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.277507 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-trusted-ca-bundle\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.277577 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.277532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-console-config\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.278235 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.278185 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-oauth-serving-cert\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.278235 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.278197 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-service-ca\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.278359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.278287 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-console-config\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.278470 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.278454 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-trusted-ca-bundle\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.279887 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.279867 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-oauth-config\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.279980 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.279963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-serving-cert\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.284894 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.284871 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgmr\" (UniqueName: \"kubernetes.io/projected/686dc483-33bd-4414-a2a2-89c0875785b7-kube-api-access-gkgmr\") pod \"console-6499575489-ln7cf\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.416100 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.416012 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:07.544053 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.544027 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6499575489-ln7cf"] Jul 22 04:35:07.546242 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:35:07.546217 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686dc483_33bd_4414_a2a2_89c0875785b7.slice/crio-58cff2e777c43fed908b43afe928b4a9ebe2e2759ab1fd509b0cbe39fc92f163 WatchSource:0}: Error finding container 58cff2e777c43fed908b43afe928b4a9ebe2e2759ab1fd509b0cbe39fc92f163: Status 404 returned error can't find the container with id 58cff2e777c43fed908b43afe928b4a9ebe2e2759ab1fd509b0cbe39fc92f163 Jul 22 04:35:07.564140 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:07.564113 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6499575489-ln7cf" event={"ID":"686dc483-33bd-4414-a2a2-89c0875785b7","Type":"ContainerStarted","Data":"58cff2e777c43fed908b43afe928b4a9ebe2e2759ab1fd509b0cbe39fc92f163"} Jul 22 04:35:10.578422 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:10.578366 2572 generic.go:356] "Generic (PLEG): container finished" podID="02c59b58-bf2a-40bb-85dd-77a54ced505c" containerID="fbd1f876001b08c3c1d097be3b790bc8cc919ff568a5d8596abb84234cd1e1d6" exitCode=0 Jul 22 04:35:10.578908 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:10.578447 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-gz2rz" event={"ID":"02c59b58-bf2a-40bb-85dd-77a54ced505c","Type":"ContainerDied","Data":"fbd1f876001b08c3c1d097be3b790bc8cc919ff568a5d8596abb84234cd1e1d6"} Jul 22 04:35:10.578984 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:10.578904 2572 scope.go:117] "RemoveContainer" containerID="fbd1f876001b08c3c1d097be3b790bc8cc919ff568a5d8596abb84234cd1e1d6" Jul 22 04:35:10.580043 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:10.580015 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6499575489-ln7cf" event={"ID":"686dc483-33bd-4414-a2a2-89c0875785b7","Type":"ContainerStarted","Data":"8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60"} Jul 22 04:35:10.613275 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:10.613229 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6499575489-ln7cf" podStartSLOduration=0.885526236 podStartE2EDuration="3.613213333s" totalImagesPullingTime="2.727687097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:35:10.612452725 +0000 UTC m=+122.331057863" watchObservedRunningTime="2026-07-22 04:35:10.613213333 +0000 UTC m=+122.331818516" Jul 22 04:35:11.586706 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:11.586668 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-gz2rz" event={"ID":"02c59b58-bf2a-40bb-85dd-77a54ced505c","Type":"ContainerStarted","Data":"4c05684d2e739cc6e4ad768db17c91d7aa56e4e20c273c1b9458cd4e9137ec23"} Jul 22 04:35:14.598612 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:14.598579 2572 generic.go:356] "Generic (PLEG): container finished" podID="8e47c0c3-83ee-4f91-b30d-83203d3d2bb5" containerID="fe23de0a9046168747df02f87ab544e05e2f43e592e5be28cec85d4e04729ba2" exitCode=0 Jul 22 04:35:14.598968 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:14.598638 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" event={"ID":"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5","Type":"ContainerDied","Data":"fe23de0a9046168747df02f87ab544e05e2f43e592e5be28cec85d4e04729ba2"} Jul 22 04:35:14.598968 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:14.598940 2572 scope.go:117] "RemoveContainer" containerID="fe23de0a9046168747df02f87ab544e05e2f43e592e5be28cec85d4e04729ba2" Jul 22 04:35:15.604863 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:15.604831 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mrht5" event={"ID":"8e47c0c3-83ee-4f91-b30d-83203d3d2bb5","Type":"ContainerStarted","Data":"bada5f0d3a48fe21ec32b6e79ac69afe63ab7621458677a30333bac56622c952"} Jul 22 04:35:17.416809 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:17.416773 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:17.417302 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:17.416828 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:17.421477 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:17.421456 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:17.616941 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:17.616903 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:35:51.046262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.046227 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:35:51.046768 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.046533 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="alertmanager" containerID="cri-o://66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032" gracePeriod=120 Jul 22 04:35:51.046768 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.046619 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy-metric" containerID="cri-o://9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9" gracePeriod=120 Jul 22 04:35:51.046768 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.046631 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="prom-label-proxy" containerID="cri-o://ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d" gracePeriod=120 Jul 22 04:35:51.046768 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.046672 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="config-reloader" containerID="cri-o://519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e" gracePeriod=120 Jul 22 04:35:51.046768 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.046682 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy" containerID="cri-o://87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37" gracePeriod=120 Jul 22 04:35:51.046768 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.046656 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy-web" containerID="cri-o://e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2" gracePeriod=120 Jul 22 04:35:51.756536 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.756500 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerID="ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d" exitCode=0 Jul 22 04:35:51.756536 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.756528 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerID="87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37" exitCode=0 Jul 22 04:35:51.756536 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.756536 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerID="519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e" exitCode=0 Jul 22 04:35:51.756536 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.756541 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerID="66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032" exitCode=0 Jul 22 04:35:51.756807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.756573 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerDied","Data":"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d"} Jul 22 04:35:51.756807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.756672 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerDied","Data":"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37"} Jul 22 04:35:51.756807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.756700 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerDied","Data":"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e"} Jul 22 04:35:51.756807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:51.756724 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerDied","Data":"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032"} Jul 22 04:35:52.287981 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.287955 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.354270 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354238 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-web-config\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354462 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354284 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-main-db\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354462 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354308 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-metrics-client-ca\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354462 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354346 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354462 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354401 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnmll\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-kube-api-access-dnmll\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354462 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354437 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354482 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-cluster-tls-config\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354523 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354550 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-tls-assets\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354580 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-out\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354610 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-main-tls\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354610 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:35:52.354730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354672 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.354730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354700 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-volume\") pod \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\" (UID: \"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2\") " Jul 22 04:35:52.355108 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354739 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:35:52.355108 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354957 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-main-db\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.355108 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.354973 2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-metrics-client-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.355262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.355131 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:35:52.358232 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.358120 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:52.358232 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.358205 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-out" (OuterVolumeSpecName: "config-out") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:35:52.358539 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.358507 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-kube-api-access-dnmll" (OuterVolumeSpecName: "kube-api-access-dnmll") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "kube-api-access-dnmll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:35:52.358710 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.358684 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-volume" (OuterVolumeSpecName: "config-volume") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:52.358817 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.358782 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:52.358953 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.358928 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:52.359232 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.359208 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:52.360264 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.360239 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:35:52.362353 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.362325 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:52.369222 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.369175 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-web-config" (OuterVolumeSpecName: "web-config") pod "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" (UID: "b1a29e76-e036-4a08-aeb3-29dfe40bf0b2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:35:52.455731 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455701 2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455731 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455728 2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-cluster-tls-config\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455731 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455739 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455749 2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-tls-assets\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455758 2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-out\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455766 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-main-tls\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455774 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455783 2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-config-volume\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455793 2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-web-config\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455801 2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.455937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.455809 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnmll\" (UniqueName: \"kubernetes.io/projected/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2-kube-api-access-dnmll\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:35:52.763523 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.763492 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerID="9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9" exitCode=0 Jul 22 04:35:52.763523 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.763520 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerID="e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2" exitCode=0 Jul 22 04:35:52.763689 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.763558 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerDied","Data":"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9"} Jul 22 04:35:52.763689 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.763571 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.763689 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.763620 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerDied","Data":"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2"} Jul 22 04:35:52.763689 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.763647 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b1a29e76-e036-4a08-aeb3-29dfe40bf0b2","Type":"ContainerDied","Data":"5195854db7355cd71dc0c2d720ea9ce10394902b1bd6c3dabec91a933ad4b5cb"} Jul 22 04:35:52.763689 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.763680 2572 scope.go:117] "RemoveContainer" containerID="ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d" Jul 22 04:35:52.772427 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.772399 2572 scope.go:117] "RemoveContainer" containerID="9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9" Jul 22 04:35:52.779501 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.779483 2572 scope.go:117] "RemoveContainer" containerID="87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37" Jul 22 04:35:52.786339 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.786291 2572 scope.go:117] "RemoveContainer" containerID="e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2" Jul 22 04:35:52.787536 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.787513 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:35:52.791713 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.791691 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:35:52.793525 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.793511 2572 scope.go:117] "RemoveContainer" containerID="519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e" Jul 22 04:35:52.801089 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.801068 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" path="/var/lib/kubelet/pods/b1a29e76-e036-4a08-aeb3-29dfe40bf0b2/volumes" Jul 22 04:35:52.801143 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.801097 2572 scope.go:117] "RemoveContainer" containerID="66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032" Jul 22 04:35:52.808458 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.808439 2572 scope.go:117] "RemoveContainer" containerID="7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e" Jul 22 04:35:52.815151 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.815115 2572 scope.go:117] "RemoveContainer" containerID="ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d" Jul 22 04:35:52.815455 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:35:52.815435 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d\": container with ID starting with ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d not found: ID does not exist" containerID="ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d" Jul 22 04:35:52.815525 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.815465 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d"} err="failed to get container status \"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d\": rpc error: code = NotFound desc = could not find container \"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d\": container with ID starting with ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d not found: ID does not exist" Jul 22 04:35:52.815525 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.815482 2572 scope.go:117] "RemoveContainer" containerID="9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9" Jul 22 04:35:52.815755 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:35:52.815736 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9\": container with ID starting with 9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9 not found: ID does not exist" containerID="9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9" Jul 22 04:35:52.815827 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.815765 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9"} err="failed to get container status \"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9\": rpc error: code = NotFound desc = could not find container \"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9\": container with ID starting with 9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9 not found: ID does not exist" Jul 22 04:35:52.815827 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.815787 2572 scope.go:117] "RemoveContainer" containerID="87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37" Jul 22 04:35:52.816040 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:35:52.816022 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37\": container with ID starting with 87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37 not found: ID does not exist" containerID="87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37" Jul 22 04:35:52.816086 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816045 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37"} err="failed to get container status \"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37\": rpc error: code = NotFound desc = could not find container \"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37\": container with ID starting with 87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37 not found: ID does not exist" Jul 22 04:35:52.816086 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816059 2572 scope.go:117] "RemoveContainer" containerID="e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2" Jul 22 04:35:52.816316 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:35:52.816291 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2\": container with ID starting with e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2 not found: ID does not exist" containerID="e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2" Jul 22 04:35:52.816368 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816326 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2"} err="failed to get container status \"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2\": rpc error: code = NotFound desc = could not find container \"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2\": container with ID starting with e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2 not found: ID does not exist" Jul 22 04:35:52.816368 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816346 2572 scope.go:117] "RemoveContainer" containerID="519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e" Jul 22 04:35:52.816542 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816528 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:35:52.816613 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:35:52.816585 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e\": container with ID starting with 519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e not found: ID does not exist" containerID="519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e" Jul 22 04:35:52.816652 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816621 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e"} err="failed to get container status \"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e\": rpc error: code = NotFound desc = could not find container \"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e\": container with ID starting with 519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e not found: ID does not exist" Jul 22 04:35:52.816652 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816641 2572 scope.go:117] "RemoveContainer" containerID="66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032" Jul 22 04:35:52.816885 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816869 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy-web" Jul 22 04:35:52.816935 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816889 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy-web" Jul 22 04:35:52.816935 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816898 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="init-config-reloader" Jul 22 04:35:52.816935 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816904 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="init-config-reloader" Jul 22 04:35:52.816935 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816912 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="alertmanager" Jul 22 04:35:52.816935 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816919 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="alertmanager" Jul 22 04:35:52.816935 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816925 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy" Jul 22 04:35:52.816935 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816930 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy" Jul 22 04:35:52.816935 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816937 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="config-reloader" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816943 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="config-reloader" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:35:52.816868 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032\": container with ID starting with 66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032 not found: ID does not exist" containerID="66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816959 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy-metric" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816967 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy-metric" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816973 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="prom-label-proxy" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816979 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="prom-label-proxy" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816973 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032"} err="failed to get container status \"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032\": rpc error: code = NotFound desc = could not find container \"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032\": container with ID starting with 66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032 not found: ID does not exist" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.816994 2572 scope.go:117] "RemoveContainer" containerID="7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817030 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="config-reloader" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817041 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy-metric" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817048 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="alertmanager" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817054 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="prom-label-proxy" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817059 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy-web" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817064 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a29e76-e036-4a08-aeb3-29dfe40bf0b2" containerName="kube-rbac-proxy" Jul 22 04:35:52.817240 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:35:52.817211 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e\": container with ID starting with 7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e not found: ID does not exist" containerID="7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e" Jul 22 04:35:52.817707 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817242 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e"} err="failed to get container status \"7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e\": rpc error: code = NotFound desc = could not find container \"7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e\": container with ID starting with 7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e not found: ID does not exist" Jul 22 04:35:52.817707 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817261 2572 scope.go:117] "RemoveContainer" containerID="ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d" Jul 22 04:35:52.817707 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817492 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d"} err="failed to get container status \"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d\": rpc error: code = NotFound desc = could not find container \"ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d\": container with ID starting with ca7af40564676cd7e22bee97331dfb10d1788a5675a22e78d28a71f5fb96f37d not found: ID does not exist" Jul 22 04:35:52.817707 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817509 2572 scope.go:117] "RemoveContainer" containerID="9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9" Jul 22 04:35:52.817707 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817693 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9"} err="failed to get container status \"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9\": rpc error: code = NotFound desc = could not find container \"9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9\": container with ID starting with 9bd53e7528c35e55c2e6dcf661076767920aefd6039882909b2f24e172ebb3b9 not found: ID does not exist" Jul 22 04:35:52.817869 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817716 2572 scope.go:117] "RemoveContainer" containerID="87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37" Jul 22 04:35:52.817961 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817943 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37"} err="failed to get container status \"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37\": rpc error: code = NotFound desc = could not find container \"87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37\": container with ID starting with 87f37395b664b66344b49d92eeefc174718f016c18e8a85b1cd1484254713d37 not found: ID does not exist" Jul 22 04:35:52.817995 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.817964 2572 scope.go:117] "RemoveContainer" containerID="e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2" Jul 22 04:35:52.818169 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.818152 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2"} err="failed to get container status \"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2\": rpc error: code = NotFound desc = could not find container \"e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2\": container with ID starting with e224ccaf165a4eec5e6eeaad08f1afdb7b5144e813723a64642c008f45aeecf2 not found: ID does not exist" Jul 22 04:35:52.818229 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.818168 2572 scope.go:117] "RemoveContainer" containerID="519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e" Jul 22 04:35:52.818410 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.818388 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e"} err="failed to get container status \"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e\": rpc error: code = NotFound desc = could not find container \"519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e\": container with ID starting with 519f546dbf9831685bde4dce547869f6eb84291ebc0b397b22b2bf73d5ec485e not found: ID does not exist" Jul 22 04:35:52.818410 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.818409 2572 scope.go:117] "RemoveContainer" containerID="66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032" Jul 22 04:35:52.818652 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.818634 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032"} err="failed to get container status \"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032\": rpc error: code = NotFound desc = could not find container \"66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032\": container with ID starting with 66377a1e35a03418678849ef4fb1728b1b8ae1c49e9b014b6c5063d807f54032 not found: ID does not exist" Jul 22 04:35:52.818706 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.818652 2572 scope.go:117] "RemoveContainer" containerID="7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e" Jul 22 04:35:52.818845 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.818826 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e"} err="failed to get container status \"7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e\": rpc error: code = NotFound desc = could not find container \"7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e\": container with ID starting with 7e85972ceffe5e73678ede9ea186b1b852ad451180e7ff40d4bd561d1332726e not found: ID does not exist" Jul 22 04:35:52.823352 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.823334 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.825953 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.825934 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 04:35:52.826064 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.826000 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 04:35:52.826064 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.826021 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 04:35:52.826456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.826295 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 04:35:52.826456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.826307 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l5qtd\"" Jul 22 04:35:52.826456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.826300 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 04:35:52.826456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.826370 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 04:35:52.826456 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.826451 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 04:35:52.826918 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.826643 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 04:35:52.831828 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.831809 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 04:35:52.832974 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.832951 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:35:52.857950 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.857906 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8323e423-1641-434c-ae24-8161a3dbb46e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858089 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.857956 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858089 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.857985 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8323e423-1641-434c-ae24-8161a3dbb46e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858089 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbn8\" (UniqueName: \"kubernetes.io/projected/8323e423-1641-434c-ae24-8161a3dbb46e-kube-api-access-pdbn8\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858089 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858045 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-web-config\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858089 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858069 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858350 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858350 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858119 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858350 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858196 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8323e423-1641-434c-ae24-8161a3dbb46e-config-out\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858350 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858236 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858350 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858350 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8323e423-1641-434c-ae24-8161a3dbb46e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.858645 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.858349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8323e423-1641-434c-ae24-8161a3dbb46e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.958929 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.958882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.958929 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.958933 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959247 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.958960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8323e423-1641-434c-ae24-8161a3dbb46e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959247 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.958988 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8323e423-1641-434c-ae24-8161a3dbb46e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959247 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959028 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8323e423-1641-434c-ae24-8161a3dbb46e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959247 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959247 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959075 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8323e423-1641-434c-ae24-8161a3dbb46e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959247 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959105 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbn8\" (UniqueName: \"kubernetes.io/projected/8323e423-1641-434c-ae24-8161a3dbb46e-kube-api-access-pdbn8\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959597 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959572 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-web-config\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959657 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959710 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959670 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959710 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959684 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8323e423-1641-434c-ae24-8161a3dbb46e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959809 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.959869 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8323e423-1641-434c-ae24-8161a3dbb46e-config-out\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.960000 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.959975 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8323e423-1641-434c-ae24-8161a3dbb46e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.960160 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.960120 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8323e423-1641-434c-ae24-8161a3dbb46e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.962176 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.962151 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.962276 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.962244 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.962322 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.962282 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.962411 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.962337 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8323e423-1641-434c-ae24-8161a3dbb46e-config-out\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.962411 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.962396 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.962567 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.962548 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.962791 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.962760 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-web-config\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.962992 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.962974 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8323e423-1641-434c-ae24-8161a3dbb46e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.964085 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.964067 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8323e423-1641-434c-ae24-8161a3dbb46e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:52.966270 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:52.966252 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbn8\" (UniqueName: \"kubernetes.io/projected/8323e423-1641-434c-ae24-8161a3dbb46e-kube-api-access-pdbn8\") pod \"alertmanager-main-0\" (UID: \"8323e423-1641-434c-ae24-8161a3dbb46e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:53.133353 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.133325 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 04:35:53.266730 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.266703 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 04:35:53.269057 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:35:53.269027 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8323e423_1641_434c_ae24_8161a3dbb46e.slice/crio-b452d7028029f399db2417a3b02be4c6e8eb706f9bb834f684bebcab9b99a27d WatchSource:0}: Error finding container b452d7028029f399db2417a3b02be4c6e8eb706f9bb834f684bebcab9b99a27d: Status 404 returned error can't find the container with id b452d7028029f399db2417a3b02be4c6e8eb706f9bb834f684bebcab9b99a27d Jul 22 04:35:53.403343 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.403270 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c97c776db-2grgr"] Jul 22 04:35:53.406843 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.406819 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.415643 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.415618 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c97c776db-2grgr"] Jul 22 04:35:53.463003 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.462969 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-config\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.463135 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.463026 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrgt\" (UniqueName: \"kubernetes.io/projected/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-kube-api-access-qdrgt\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.463135 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.463053 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-oauth-config\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.463135 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.463085 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-serving-cert\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.463135 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.463101 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-trusted-ca-bundle\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.463253 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.463142 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-service-ca\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.463253 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.463176 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-oauth-serving-cert\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.563760 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.563724 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-oauth-serving-cert\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.563908 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.563771 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-config\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.563908 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.563812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrgt\" (UniqueName: \"kubernetes.io/projected/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-kube-api-access-qdrgt\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.563908 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.563838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-oauth-config\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.563908 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.563856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-serving-cert\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.563908 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.563873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-trusted-ca-bundle\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.563908 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.563896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-service-ca\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.564531 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.564506 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-config\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.564686 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.564570 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-oauth-serving-cert\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.564686 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.564650 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-service-ca\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.564889 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.564854 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-trusted-ca-bundle\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.566340 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.566309 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-oauth-config\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.566452 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.566422 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-serving-cert\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.571816 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.571796 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrgt\" (UniqueName: \"kubernetes.io/projected/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-kube-api-access-qdrgt\") pod \"console-6c97c776db-2grgr\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.717085 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.717006 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:35:53.772425 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.771691 2572 generic.go:356] "Generic (PLEG): container finished" podID="8323e423-1641-434c-ae24-8161a3dbb46e" containerID="2262ff4739d1c5e94bc12510b07782c52cac0b533edec1589713e06c2ee79860" exitCode=0 Jul 22 04:35:53.772425 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.771757 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8323e423-1641-434c-ae24-8161a3dbb46e","Type":"ContainerDied","Data":"2262ff4739d1c5e94bc12510b07782c52cac0b533edec1589713e06c2ee79860"} Jul 22 04:35:53.772425 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.771828 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8323e423-1641-434c-ae24-8161a3dbb46e","Type":"ContainerStarted","Data":"b452d7028029f399db2417a3b02be4c6e8eb706f9bb834f684bebcab9b99a27d"} Jul 22 04:35:53.854361 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:53.854341 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c97c776db-2grgr"] Jul 22 04:35:53.857064 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:35:53.857038 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc82bd7_3e98_4cf8_bc7f_373a4709f95f.slice/crio-a8b1b5b423ef0706bc08b261b4a88144ddc87b6063de15379bd937952389d1de WatchSource:0}: Error finding container a8b1b5b423ef0706bc08b261b4a88144ddc87b6063de15379bd937952389d1de: Status 404 returned error can't find the container with id a8b1b5b423ef0706bc08b261b4a88144ddc87b6063de15379bd937952389d1de Jul 22 04:35:54.777168 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.777123 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97c776db-2grgr" event={"ID":"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f","Type":"ContainerStarted","Data":"7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a"} Jul 22 04:35:54.777610 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.777188 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97c776db-2grgr" event={"ID":"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f","Type":"ContainerStarted","Data":"a8b1b5b423ef0706bc08b261b4a88144ddc87b6063de15379bd937952389d1de"} Jul 22 04:35:54.780235 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.780207 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8323e423-1641-434c-ae24-8161a3dbb46e","Type":"ContainerStarted","Data":"d2a52b4c4c8f11a1c5f27704657e0fca331603fc62aab8b64fbe7722ad046d2b"} Jul 22 04:35:54.780348 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.780264 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8323e423-1641-434c-ae24-8161a3dbb46e","Type":"ContainerStarted","Data":"7ea28c9668f67744cadf3a7322b75a85d85f47f49a90eea4939f82ac9a1cf5e8"} Jul 22 04:35:54.780348 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.780291 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8323e423-1641-434c-ae24-8161a3dbb46e","Type":"ContainerStarted","Data":"107130e1db934517f0594767581a9372b13efea6d74b089784d45cc5c3213bbe"} Jul 22 04:35:54.780348 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.780315 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8323e423-1641-434c-ae24-8161a3dbb46e","Type":"ContainerStarted","Data":"26aacd2138ecace1f0e6e4ae02b1f0794c19bfc6c0e17e62c5e14657b25359a5"} Jul 22 04:35:54.780348 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.780341 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8323e423-1641-434c-ae24-8161a3dbb46e","Type":"ContainerStarted","Data":"8540b2f55d046bf0534d70e16c681104a2fdec1ed6f673e492d5aac3ce7d417e"} Jul 22 04:35:54.780517 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.780363 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8323e423-1641-434c-ae24-8161a3dbb46e","Type":"ContainerStarted","Data":"f6ebd6f5edf9761033ad9aa6745074a4d42a564d099ab72e0cc19b75a5e90da4"} Jul 22 04:35:54.796992 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.796940 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c97c776db-2grgr" podStartSLOduration=1.79692143 podStartE2EDuration="1.79692143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:35:54.794867526 +0000 UTC m=+166.513472693" watchObservedRunningTime="2026-07-22 04:35:54.79692143 +0000 UTC m=+166.515526566" Jul 22 04:35:54.822615 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:35:54.822560 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.806188359 podStartE2EDuration="2.822540188s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.351829ms" isStatefulPod=false podCreationTimestamp="2026-07-22 04:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:35:54.820052367 +0000 UTC m=+166.538657502" watchObservedRunningTime="2026-07-22 04:35:54.822540188 +0000 UTC m=+166.541145327" Jul 22 04:36:03.717882 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:03.717775 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:36:03.717882 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:03.717868 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:36:03.722734 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:03.722710 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:36:03.819639 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:03.819611 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:36:03.863888 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:03.863855 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6499575489-ln7cf"] Jul 22 04:36:28.893503 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:28.893443 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6499575489-ln7cf" podUID="686dc483-33bd-4414-a2a2-89c0875785b7" containerName="console" containerID="cri-o://8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60" gracePeriod=15 Jul 22 04:36:29.130815 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.130791 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:36:29.169246 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169184 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-oauth-serving-cert\") pod \"686dc483-33bd-4414-a2a2-89c0875785b7\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " Jul 22 04:36:29.169246 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169227 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-service-ca\") pod \"686dc483-33bd-4414-a2a2-89c0875785b7\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " Jul 22 04:36:29.169452 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169253 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-oauth-config\") pod \"686dc483-33bd-4414-a2a2-89c0875785b7\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " Jul 22 04:36:29.169452 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169324 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-trusted-ca-bundle\") pod \"686dc483-33bd-4414-a2a2-89c0875785b7\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " Jul 22 04:36:29.169452 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169351 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkgmr\" (UniqueName: \"kubernetes.io/projected/686dc483-33bd-4414-a2a2-89c0875785b7-kube-api-access-gkgmr\") pod \"686dc483-33bd-4414-a2a2-89c0875785b7\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " Jul 22 04:36:29.169452 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169402 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-serving-cert\") pod \"686dc483-33bd-4414-a2a2-89c0875785b7\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " Jul 22 04:36:29.169452 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169439 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-console-config\") pod \"686dc483-33bd-4414-a2a2-89c0875785b7\" (UID: \"686dc483-33bd-4414-a2a2-89c0875785b7\") " Jul 22 04:36:29.169807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169608 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "686dc483-33bd-4414-a2a2-89c0875785b7" (UID: "686dc483-33bd-4414-a2a2-89c0875785b7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:36:29.169807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169632 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-service-ca" (OuterVolumeSpecName: "service-ca") pod "686dc483-33bd-4414-a2a2-89c0875785b7" (UID: "686dc483-33bd-4414-a2a2-89c0875785b7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:36:29.169807 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169723 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "686dc483-33bd-4414-a2a2-89c0875785b7" (UID: "686dc483-33bd-4414-a2a2-89c0875785b7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:36:29.169972 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169747 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-oauth-serving-cert\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:36:29.169972 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169870 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:36:29.169972 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.169940 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-console-config" (OuterVolumeSpecName: "console-config") pod "686dc483-33bd-4414-a2a2-89c0875785b7" (UID: "686dc483-33bd-4414-a2a2-89c0875785b7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:36:29.171600 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.171578 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "686dc483-33bd-4414-a2a2-89c0875785b7" (UID: "686dc483-33bd-4414-a2a2-89c0875785b7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:36:29.171697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.171599 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "686dc483-33bd-4414-a2a2-89c0875785b7" (UID: "686dc483-33bd-4414-a2a2-89c0875785b7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:36:29.171697 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.171604 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686dc483-33bd-4414-a2a2-89c0875785b7-kube-api-access-gkgmr" (OuterVolumeSpecName: "kube-api-access-gkgmr") pod "686dc483-33bd-4414-a2a2-89c0875785b7" (UID: "686dc483-33bd-4414-a2a2-89c0875785b7"). InnerVolumeSpecName "kube-api-access-gkgmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:36:29.270281 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.270261 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-oauth-config\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:36:29.270392 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.270286 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-trusted-ca-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:36:29.270392 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.270301 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkgmr\" (UniqueName: \"kubernetes.io/projected/686dc483-33bd-4414-a2a2-89c0875785b7-kube-api-access-gkgmr\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:36:29.270392 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.270313 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/686dc483-33bd-4414-a2a2-89c0875785b7-console-serving-cert\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:36:29.270392 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.270325 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/686dc483-33bd-4414-a2a2-89c0875785b7-console-config\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:36:29.916971 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.916940 2572 generic.go:356] "Generic (PLEG): container finished" podID="686dc483-33bd-4414-a2a2-89c0875785b7" containerID="8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60" exitCode=0 Jul 22 04:36:29.917429 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.916985 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6499575489-ln7cf" Jul 22 04:36:29.917429 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.917035 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6499575489-ln7cf" event={"ID":"686dc483-33bd-4414-a2a2-89c0875785b7","Type":"ContainerDied","Data":"8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60"} Jul 22 04:36:29.917429 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.917096 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6499575489-ln7cf" event={"ID":"686dc483-33bd-4414-a2a2-89c0875785b7","Type":"ContainerDied","Data":"58cff2e777c43fed908b43afe928b4a9ebe2e2759ab1fd509b0cbe39fc92f163"} Jul 22 04:36:29.917429 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.917129 2572 scope.go:117] "RemoveContainer" containerID="8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60" Jul 22 04:36:29.925950 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.925932 2572 scope.go:117] "RemoveContainer" containerID="8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60" Jul 22 04:36:29.926189 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:36:29.926171 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60\": container with ID starting with 8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60 not found: ID does not exist" containerID="8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60" Jul 22 04:36:29.926257 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.926195 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60"} err="failed to get container status \"8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60\": rpc error: code = NotFound desc = could not find container \"8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60\": container with ID starting with 8318b84be3549372f9f4986a105dfe7fe7ff31305ed5a6788afb022e61e3cd60 not found: ID does not exist" Jul 22 04:36:29.938060 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.938014 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6499575489-ln7cf"] Jul 22 04:36:29.939518 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:29.939499 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6499575489-ln7cf"] Jul 22 04:36:30.800107 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:30.800071 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="686dc483-33bd-4414-a2a2-89c0875785b7" path="/var/lib/kubelet/pods/686dc483-33bd-4414-a2a2-89c0875785b7/volumes" Jul 22 04:36:44.908087 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.908048 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k"] Jul 22 04:36:44.908473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.908411 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686dc483-33bd-4414-a2a2-89c0875785b7" containerName="console" Jul 22 04:36:44.908473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.908422 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="686dc483-33bd-4414-a2a2-89c0875785b7" containerName="console" Jul 22 04:36:44.908545 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.908492 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="686dc483-33bd-4414-a2a2-89c0875785b7" containerName="console" Jul 22 04:36:44.912809 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.912791 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:44.915019 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.914998 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 04:36:44.915897 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.915879 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jk8wp\"" Jul 22 04:36:44.915987 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.915934 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 04:36:44.921127 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.921106 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k"] Jul 22 04:36:44.991597 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.991566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:44.991738 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.991611 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qx54\" (UniqueName: \"kubernetes.io/projected/991baf4e-10dd-4105-8e71-d523e14d02fb-kube-api-access-8qx54\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:44.991738 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:44.991701 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:45.092694 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.092668 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:45.092824 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.092713 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qx54\" (UniqueName: \"kubernetes.io/projected/991baf4e-10dd-4105-8e71-d523e14d02fb-kube-api-access-8qx54\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:45.092824 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.092759 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:45.093137 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.093115 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:45.093182 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.093166 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:45.100620 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.100590 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qx54\" (UniqueName: \"kubernetes.io/projected/991baf4e-10dd-4105-8e71-d523e14d02fb-kube-api-access-8qx54\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:45.222662 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.222571 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:36:45.347798 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.347774 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k"] Jul 22 04:36:45.350359 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:36:45.350332 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991baf4e_10dd_4105_8e71_d523e14d02fb.slice/crio-538d768b57bc15ad938bb0b943e3888fe82f54f05c77f3119e410c09bd0aad06 WatchSource:0}: Error finding container 538d768b57bc15ad938bb0b943e3888fe82f54f05c77f3119e410c09bd0aad06: Status 404 returned error can't find the container with id 538d768b57bc15ad938bb0b943e3888fe82f54f05c77f3119e410c09bd0aad06 Jul 22 04:36:45.979054 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:45.979013 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" event={"ID":"991baf4e-10dd-4105-8e71-d523e14d02fb","Type":"ContainerStarted","Data":"538d768b57bc15ad938bb0b943e3888fe82f54f05c77f3119e410c09bd0aad06"} Jul 22 04:36:52.010359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:52.010318 2572 generic.go:356] "Generic (PLEG): container finished" podID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerID="e3fd8863c1e7fb47f94ab91e331282c3ac543244f317431016aad1716864ad08" exitCode=0 Jul 22 04:36:52.010727 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:52.010408 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" event={"ID":"991baf4e-10dd-4105-8e71-d523e14d02fb","Type":"ContainerDied","Data":"e3fd8863c1e7fb47f94ab91e331282c3ac543244f317431016aad1716864ad08"} Jul 22 04:36:55.024718 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:55.024686 2572 generic.go:356] "Generic (PLEG): container finished" podID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerID="45c80866cbe0e4302efa2ef4ec48ae8b6a3cdbfc3bbaebd903987cbcb0b8b04f" exitCode=0 Jul 22 04:36:55.025023 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:36:55.024767 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" event={"ID":"991baf4e-10dd-4105-8e71-d523e14d02fb","Type":"ContainerDied","Data":"45c80866cbe0e4302efa2ef4ec48ae8b6a3cdbfc3bbaebd903987cbcb0b8b04f"} Jul 22 04:37:02.058273 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:02.058231 2572 generic.go:356] "Generic (PLEG): container finished" podID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerID="5bc3e69a507dc04fb43081946db3c05520744bacd8d93ae054e3f345a28e7f9e" exitCode=0 Jul 22 04:37:02.058711 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:02.058320 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" event={"ID":"991baf4e-10dd-4105-8e71-d523e14d02fb","Type":"ContainerDied","Data":"5bc3e69a507dc04fb43081946db3c05520744bacd8d93ae054e3f345a28e7f9e"} Jul 22 04:37:03.185321 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.185298 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:37:03.351841 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.351751 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qx54\" (UniqueName: \"kubernetes.io/projected/991baf4e-10dd-4105-8e71-d523e14d02fb-kube-api-access-8qx54\") pod \"991baf4e-10dd-4105-8e71-d523e14d02fb\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " Jul 22 04:37:03.351841 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.351806 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-util\") pod \"991baf4e-10dd-4105-8e71-d523e14d02fb\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " Jul 22 04:37:03.352008 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.351880 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-bundle\") pod \"991baf4e-10dd-4105-8e71-d523e14d02fb\" (UID: \"991baf4e-10dd-4105-8e71-d523e14d02fb\") " Jul 22 04:37:03.352535 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.352500 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-bundle" (OuterVolumeSpecName: "bundle") pod "991baf4e-10dd-4105-8e71-d523e14d02fb" (UID: "991baf4e-10dd-4105-8e71-d523e14d02fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:37:03.353899 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.353874 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991baf4e-10dd-4105-8e71-d523e14d02fb-kube-api-access-8qx54" (OuterVolumeSpecName: "kube-api-access-8qx54") pod "991baf4e-10dd-4105-8e71-d523e14d02fb" (UID: "991baf4e-10dd-4105-8e71-d523e14d02fb"). InnerVolumeSpecName "kube-api-access-8qx54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:37:03.355919 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.355890 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-util" (OuterVolumeSpecName: "util") pod "991baf4e-10dd-4105-8e71-d523e14d02fb" (UID: "991baf4e-10dd-4105-8e71-d523e14d02fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:37:03.452324 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.452300 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-util\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:03.452324 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.452321 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/991baf4e-10dd-4105-8e71-d523e14d02fb-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:03.452481 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:03.452330 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qx54\" (UniqueName: \"kubernetes.io/projected/991baf4e-10dd-4105-8e71-d523e14d02fb-kube-api-access-8qx54\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:04.067931 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:04.067892 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" event={"ID":"991baf4e-10dd-4105-8e71-d523e14d02fb","Type":"ContainerDied","Data":"538d768b57bc15ad938bb0b943e3888fe82f54f05c77f3119e410c09bd0aad06"} Jul 22 04:37:04.068103 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:04.067943 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538d768b57bc15ad938bb0b943e3888fe82f54f05c77f3119e410c09bd0aad06" Jul 22 04:37:04.068103 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:04.067911 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q286k" Jul 22 04:37:07.815678 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.815643 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw"] Jul 22 04:37:07.816038 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.815980 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerName="pull" Jul 22 04:37:07.816038 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.815992 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerName="pull" Jul 22 04:37:07.816038 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.816003 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerName="util" Jul 22 04:37:07.816038 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.816009 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerName="util" Jul 22 04:37:07.816038 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.816019 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerName="extract" Jul 22 04:37:07.816038 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.816025 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerName="extract" Jul 22 04:37:07.816217 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.816086 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="991baf4e-10dd-4105-8e71-d523e14d02fb" containerName="extract" Jul 22 04:37:07.866057 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.866020 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw"] Jul 22 04:37:07.866206 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.866111 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" Jul 22 04:37:07.868551 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.868527 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:37:07.868685 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.868569 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vz7mw\"" Jul 22 04:37:07.868685 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.868627 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 04:37:07.989718 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.989688 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklct\" (UniqueName: \"kubernetes.io/projected/02550e95-f8df-4275-bfa4-c5685d7cd0b2-kube-api-access-tklct\") pod \"cert-manager-operator-controller-manager-55c444bf86-2ltqw\" (UID: \"02550e95-f8df-4275-bfa4-c5685d7cd0b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" Jul 22 04:37:07.989877 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:07.989729 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02550e95-f8df-4275-bfa4-c5685d7cd0b2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2ltqw\" (UID: \"02550e95-f8df-4275-bfa4-c5685d7cd0b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" Jul 22 04:37:08.090564 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:08.090489 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklct\" (UniqueName: \"kubernetes.io/projected/02550e95-f8df-4275-bfa4-c5685d7cd0b2-kube-api-access-tklct\") pod \"cert-manager-operator-controller-manager-55c444bf86-2ltqw\" (UID: \"02550e95-f8df-4275-bfa4-c5685d7cd0b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" Jul 22 04:37:08.090564 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:08.090526 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02550e95-f8df-4275-bfa4-c5685d7cd0b2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2ltqw\" (UID: \"02550e95-f8df-4275-bfa4-c5685d7cd0b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" Jul 22 04:37:08.090858 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:08.090842 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02550e95-f8df-4275-bfa4-c5685d7cd0b2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2ltqw\" (UID: \"02550e95-f8df-4275-bfa4-c5685d7cd0b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" Jul 22 04:37:08.098191 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:08.098164 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklct\" (UniqueName: \"kubernetes.io/projected/02550e95-f8df-4275-bfa4-c5685d7cd0b2-kube-api-access-tklct\") pod \"cert-manager-operator-controller-manager-55c444bf86-2ltqw\" (UID: \"02550e95-f8df-4275-bfa4-c5685d7cd0b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" Jul 22 04:37:08.175106 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:08.175080 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" Jul 22 04:37:08.328332 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:08.328289 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw"] Jul 22 04:37:08.332842 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:37:08.332809 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02550e95_f8df_4275_bfa4_c5685d7cd0b2.slice/crio-6800232517f130c003d59a05939eb73acd393a5c35bf098454af8e04f4d9cc4d WatchSource:0}: Error finding container 6800232517f130c003d59a05939eb73acd393a5c35bf098454af8e04f4d9cc4d: Status 404 returned error can't find the container with id 6800232517f130c003d59a05939eb73acd393a5c35bf098454af8e04f4d9cc4d Jul 22 04:37:09.087547 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:09.087508 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" event={"ID":"02550e95-f8df-4275-bfa4-c5685d7cd0b2","Type":"ContainerStarted","Data":"6800232517f130c003d59a05939eb73acd393a5c35bf098454af8e04f4d9cc4d"} Jul 22 04:37:11.098442 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.098410 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" event={"ID":"02550e95-f8df-4275-bfa4-c5685d7cd0b2","Type":"ContainerStarted","Data":"05d171e58dd92cd8bde8e367d860413dc79bbb9b2dea525b39b86ea86ea0f35b"} Jul 22 04:37:11.118208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.118151 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2ltqw" podStartSLOduration=1.68983398 podStartE2EDuration="4.118135093s" totalImagesPullingTime="2.428301113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:37:11.115705178 +0000 UTC m=+242.834310314" watchObservedRunningTime="2026-07-22 04:37:11.118135093 +0000 UTC m=+242.836740229" Jul 22 04:37:11.746600 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.746562 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl"] Jul 22 04:37:11.764264 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.764227 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl"] Jul 22 04:37:11.764476 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.764365 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:11.766985 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.766957 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 04:37:11.767120 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.767072 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 04:37:11.767186 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.767147 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jk8wp\"" Jul 22 04:37:11.924453 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.924414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k82tm\" (UniqueName: \"kubernetes.io/projected/bede007d-6712-4f74-b80e-bf7ac2bd92dc-kube-api-access-k82tm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:11.924647 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.924464 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:11.924647 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:11.924565 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:12.025735 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:12.025698 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k82tm\" (UniqueName: \"kubernetes.io/projected/bede007d-6712-4f74-b80e-bf7ac2bd92dc-kube-api-access-k82tm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:12.025907 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:12.025748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:12.025907 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:12.025789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:12.026211 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:12.026190 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:12.026248 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:12.026211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:12.035986 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:12.035959 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k82tm\" (UniqueName: \"kubernetes.io/projected/bede007d-6712-4f74-b80e-bf7ac2bd92dc-kube-api-access-k82tm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:12.074775 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:12.074747 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:12.224124 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:12.224098 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl"] Jul 22 04:37:12.226207 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:37:12.226167 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbede007d_6712_4f74_b80e_bf7ac2bd92dc.slice/crio-e63725a7ff94c0a75ce73f531ddb12dd3e3cfa9ecc276c20913f9e5f9a6a407a WatchSource:0}: Error finding container e63725a7ff94c0a75ce73f531ddb12dd3e3cfa9ecc276c20913f9e5f9a6a407a: Status 404 returned error can't find the container with id e63725a7ff94c0a75ce73f531ddb12dd3e3cfa9ecc276c20913f9e5f9a6a407a Jul 22 04:37:13.110330 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.110292 2572 generic.go:356] "Generic (PLEG): container finished" podID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerID="a151dfc36b842066142dedce084ba9e29b9550b51264a1955c472ad9a9b37940" exitCode=0 Jul 22 04:37:13.110507 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.110397 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" event={"ID":"bede007d-6712-4f74-b80e-bf7ac2bd92dc","Type":"ContainerDied","Data":"a151dfc36b842066142dedce084ba9e29b9550b51264a1955c472ad9a9b37940"} Jul 22 04:37:13.110507 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.110452 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" event={"ID":"bede007d-6712-4f74-b80e-bf7ac2bd92dc","Type":"ContainerStarted","Data":"e63725a7ff94c0a75ce73f531ddb12dd3e3cfa9ecc276c20913f9e5f9a6a407a"} Jul 22 04:37:13.330546 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.330458 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vz2td"] Jul 22 04:37:13.347970 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.347938 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vz2td"] Jul 22 04:37:13.348096 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.348034 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:13.350231 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.350209 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 04:37:13.350368 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.350328 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 04:37:13.350639 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.350617 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wh7dt\"" Jul 22 04:37:13.437359 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.437327 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b80427a-40db-4f74-87df-f8307fddf8be-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vz2td\" (UID: \"1b80427a-40db-4f74-87df-f8307fddf8be\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:13.437533 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.437458 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvllz\" (UniqueName: \"kubernetes.io/projected/1b80427a-40db-4f74-87df-f8307fddf8be-kube-api-access-pvllz\") pod \"cert-manager-webhook-6486ffd997-vz2td\" (UID: \"1b80427a-40db-4f74-87df-f8307fddf8be\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:13.538505 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.538469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b80427a-40db-4f74-87df-f8307fddf8be-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vz2td\" (UID: \"1b80427a-40db-4f74-87df-f8307fddf8be\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:13.538681 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.538632 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvllz\" (UniqueName: \"kubernetes.io/projected/1b80427a-40db-4f74-87df-f8307fddf8be-kube-api-access-pvllz\") pod \"cert-manager-webhook-6486ffd997-vz2td\" (UID: \"1b80427a-40db-4f74-87df-f8307fddf8be\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:13.546805 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.546774 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1b80427a-40db-4f74-87df-f8307fddf8be-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vz2td\" (UID: \"1b80427a-40db-4f74-87df-f8307fddf8be\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:13.546969 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.546948 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvllz\" (UniqueName: \"kubernetes.io/projected/1b80427a-40db-4f74-87df-f8307fddf8be-kube-api-access-pvllz\") pod \"cert-manager-webhook-6486ffd997-vz2td\" (UID: \"1b80427a-40db-4f74-87df-f8307fddf8be\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:13.679172 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.679086 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:13.828521 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:13.828486 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vz2td"] Jul 22 04:37:13.830834 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:37:13.830809 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b80427a_40db_4f74_87df_f8307fddf8be.slice/crio-a433f95e7e08c236f62cca5b3783a55a4c423392520031adeea506aeebfceee4 WatchSource:0}: Error finding container a433f95e7e08c236f62cca5b3783a55a4c423392520031adeea506aeebfceee4: Status 404 returned error can't find the container with id a433f95e7e08c236f62cca5b3783a55a4c423392520031adeea506aeebfceee4 Jul 22 04:37:14.117041 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:14.116996 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" event={"ID":"1b80427a-40db-4f74-87df-f8307fddf8be","Type":"ContainerStarted","Data":"a433f95e7e08c236f62cca5b3783a55a4c423392520031adeea506aeebfceee4"} Jul 22 04:37:16.127308 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:16.127273 2572 generic.go:356] "Generic (PLEG): container finished" podID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerID="bac828bf123ffeb19d5cd3e103dd0f96fd709ddf6f5925fcf65804def77e5ea8" exitCode=0 Jul 22 04:37:16.127736 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:16.127356 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" event={"ID":"bede007d-6712-4f74-b80e-bf7ac2bd92dc","Type":"ContainerDied","Data":"bac828bf123ffeb19d5cd3e103dd0f96fd709ddf6f5925fcf65804def77e5ea8"} Jul 22 04:37:17.133820 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:17.133781 2572 generic.go:356] "Generic (PLEG): container finished" podID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerID="c24f252dea6fe4bcf537171b934103c48fe30697e9c6d86b0402348b492d993a" exitCode=0 Jul 22 04:37:17.134252 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:17.133866 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" event={"ID":"bede007d-6712-4f74-b80e-bf7ac2bd92dc","Type":"ContainerDied","Data":"c24f252dea6fe4bcf537171b934103c48fe30697e9c6d86b0402348b492d993a"} Jul 22 04:37:18.264718 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.264690 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:18.380636 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.380601 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-bundle\") pod \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " Jul 22 04:37:18.380795 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.380693 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-util\") pod \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " Jul 22 04:37:18.380795 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.380763 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k82tm\" (UniqueName: \"kubernetes.io/projected/bede007d-6712-4f74-b80e-bf7ac2bd92dc-kube-api-access-k82tm\") pod \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\" (UID: \"bede007d-6712-4f74-b80e-bf7ac2bd92dc\") " Jul 22 04:37:18.381063 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.381032 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-bundle" (OuterVolumeSpecName: "bundle") pod "bede007d-6712-4f74-b80e-bf7ac2bd92dc" (UID: "bede007d-6712-4f74-b80e-bf7ac2bd92dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:37:18.382766 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.382739 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bede007d-6712-4f74-b80e-bf7ac2bd92dc-kube-api-access-k82tm" (OuterVolumeSpecName: "kube-api-access-k82tm") pod "bede007d-6712-4f74-b80e-bf7ac2bd92dc" (UID: "bede007d-6712-4f74-b80e-bf7ac2bd92dc"). InnerVolumeSpecName "kube-api-access-k82tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:37:18.385126 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.385101 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-util" (OuterVolumeSpecName: "util") pod "bede007d-6712-4f74-b80e-bf7ac2bd92dc" (UID: "bede007d-6712-4f74-b80e-bf7ac2bd92dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:37:18.482154 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.482075 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-util\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:18.482154 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.482106 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k82tm\" (UniqueName: \"kubernetes.io/projected/bede007d-6712-4f74-b80e-bf7ac2bd92dc-kube-api-access-k82tm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:18.482154 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:18.482122 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bede007d-6712-4f74-b80e-bf7ac2bd92dc-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:19.144276 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:19.144188 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" event={"ID":"bede007d-6712-4f74-b80e-bf7ac2bd92dc","Type":"ContainerDied","Data":"e63725a7ff94c0a75ce73f531ddb12dd3e3cfa9ecc276c20913f9e5f9a6a407a"} Jul 22 04:37:19.144276 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:19.144246 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63725a7ff94c0a75ce73f531ddb12dd3e3cfa9ecc276c20913f9e5f9a6a407a" Jul 22 04:37:19.144276 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:19.144201 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcjgxl" Jul 22 04:37:23.885631 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.885594 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lnkgf"] Jul 22 04:37:23.886397 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.886350 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerName="util" Jul 22 04:37:23.886397 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.886394 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerName="util" Jul 22 04:37:23.886553 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.886408 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerName="pull" Jul 22 04:37:23.886553 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.886418 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerName="pull" Jul 22 04:37:23.886553 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.886445 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerName="extract" Jul 22 04:37:23.886553 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.886453 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerName="extract" Jul 22 04:37:23.886725 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.886619 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bede007d-6712-4f74-b80e-bf7ac2bd92dc" containerName="extract" Jul 22 04:37:23.890346 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.890307 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lnkgf" Jul 22 04:37:23.892640 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.892596 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6hdsx\"" Jul 22 04:37:23.895527 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.895506 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lnkgf"] Jul 22 04:37:23.922418 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.922396 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfhq\" (UniqueName: \"kubernetes.io/projected/b6232e06-0f40-4ff8-bec3-783bf32bdfda-kube-api-access-vvfhq\") pod \"cert-manager-55b4d954d5-lnkgf\" (UID: \"b6232e06-0f40-4ff8-bec3-783bf32bdfda\") " pod="cert-manager/cert-manager-55b4d954d5-lnkgf" Jul 22 04:37:23.922512 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:23.922436 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6232e06-0f40-4ff8-bec3-783bf32bdfda-bound-sa-token\") pod \"cert-manager-55b4d954d5-lnkgf\" (UID: \"b6232e06-0f40-4ff8-bec3-783bf32bdfda\") " pod="cert-manager/cert-manager-55b4d954d5-lnkgf" Jul 22 04:37:24.023026 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.022995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfhq\" (UniqueName: \"kubernetes.io/projected/b6232e06-0f40-4ff8-bec3-783bf32bdfda-kube-api-access-vvfhq\") pod \"cert-manager-55b4d954d5-lnkgf\" (UID: \"b6232e06-0f40-4ff8-bec3-783bf32bdfda\") " pod="cert-manager/cert-manager-55b4d954d5-lnkgf" Jul 22 04:37:24.023175 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.023044 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6232e06-0f40-4ff8-bec3-783bf32bdfda-bound-sa-token\") pod \"cert-manager-55b4d954d5-lnkgf\" (UID: \"b6232e06-0f40-4ff8-bec3-783bf32bdfda\") " pod="cert-manager/cert-manager-55b4d954d5-lnkgf" Jul 22 04:37:24.032218 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.032187 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6232e06-0f40-4ff8-bec3-783bf32bdfda-bound-sa-token\") pod \"cert-manager-55b4d954d5-lnkgf\" (UID: \"b6232e06-0f40-4ff8-bec3-783bf32bdfda\") " pod="cert-manager/cert-manager-55b4d954d5-lnkgf" Jul 22 04:37:24.032472 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.032450 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfhq\" (UniqueName: \"kubernetes.io/projected/b6232e06-0f40-4ff8-bec3-783bf32bdfda-kube-api-access-vvfhq\") pod \"cert-manager-55b4d954d5-lnkgf\" (UID: \"b6232e06-0f40-4ff8-bec3-783bf32bdfda\") " pod="cert-manager/cert-manager-55b4d954d5-lnkgf" Jul 22 04:37:24.169312 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.169233 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" event={"ID":"1b80427a-40db-4f74-87df-f8307fddf8be","Type":"ContainerStarted","Data":"f9ba89d376e531b34a2bf91824ba91b8840fadbd6daa137b06b02049e77ed793"} Jul 22 04:37:24.169312 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.169293 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:24.185896 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.185852 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" podStartSLOduration=1.7517039620000001 podStartE2EDuration="11.185836812s" totalImagesPullingTime="9.43413285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:37:24.183881446 +0000 UTC m=+255.902486584" watchObservedRunningTime="2026-07-22 04:37:24.185836812 +0000 UTC m=+255.904441947" Jul 22 04:37:24.201352 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.201304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lnkgf" Jul 22 04:37:24.325964 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:24.325937 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lnkgf"] Jul 22 04:37:24.327725 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:37:24.327698 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6232e06_0f40_4ff8_bec3_783bf32bdfda.slice/crio-4ed30b645a12092a7f35a77967a97620d677f9988e3b8e64a70d75a9dbe72af5 WatchSource:0}: Error finding container 4ed30b645a12092a7f35a77967a97620d677f9988e3b8e64a70d75a9dbe72af5: Status 404 returned error can't find the container with id 4ed30b645a12092a7f35a77967a97620d677f9988e3b8e64a70d75a9dbe72af5 Jul 22 04:37:25.174769 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:25.174732 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lnkgf" event={"ID":"b6232e06-0f40-4ff8-bec3-783bf32bdfda","Type":"ContainerStarted","Data":"d871c47e07c3ef0d7b39c2cbb58aa01d717a58d00cfae8d2b77596925e1acae8"} Jul 22 04:37:25.175147 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:25.174776 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lnkgf" event={"ID":"b6232e06-0f40-4ff8-bec3-783bf32bdfda","Type":"ContainerStarted","Data":"4ed30b645a12092a7f35a77967a97620d677f9988e3b8e64a70d75a9dbe72af5"} Jul 22 04:37:25.190332 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:25.190280 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lnkgf" podStartSLOduration=2.190265454 podStartE2EDuration="2.190265454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:37:25.188534239 +0000 UTC m=+256.907139353" watchObservedRunningTime="2026-07-22 04:37:25.190265454 +0000 UTC m=+256.908870588" Jul 22 04:37:30.177315 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:30.177282 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vz2td" Jul 22 04:37:31.796647 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.796614 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7"] Jul 22 04:37:31.799711 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.799695 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.801951 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.801930 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 04:37:31.802084 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.801933 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 04:37:31.802686 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.802672 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jk8wp\"" Jul 22 04:37:31.806923 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.806902 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7"] Jul 22 04:37:31.884728 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.884697 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.884861 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.884741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsmt\" (UniqueName: \"kubernetes.io/projected/7a18c167-5511-43ba-a3d8-0091015bfd7b-kube-api-access-zhsmt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.884861 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.884847 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.986079 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.986043 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.986241 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.986114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.986241 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.986163 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsmt\" (UniqueName: \"kubernetes.io/projected/7a18c167-5511-43ba-a3d8-0091015bfd7b-kube-api-access-zhsmt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.986564 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.986542 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.986617 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.986552 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:31.994238 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:31.994211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsmt\" (UniqueName: \"kubernetes.io/projected/7a18c167-5511-43ba-a3d8-0091015bfd7b-kube-api-access-zhsmt\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:32.109532 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:32.109452 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:32.236621 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:32.236599 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7"] Jul 22 04:37:32.238859 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:37:32.238823 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a18c167_5511_43ba_a3d8_0091015bfd7b.slice/crio-316f0f305394600fcaad05465f3de86ccd236c881bde7619218727c155e7b3d1 WatchSource:0}: Error finding container 316f0f305394600fcaad05465f3de86ccd236c881bde7619218727c155e7b3d1: Status 404 returned error can't find the container with id 316f0f305394600fcaad05465f3de86ccd236c881bde7619218727c155e7b3d1 Jul 22 04:37:33.211264 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:33.211236 2572 generic.go:356] "Generic (PLEG): container finished" podID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerID="05f52ff477dabfc65f4097a46befbf384c0cbb1c34c3583a560996c828162fb4" exitCode=0 Jul 22 04:37:33.211681 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:33.211272 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" event={"ID":"7a18c167-5511-43ba-a3d8-0091015bfd7b","Type":"ContainerDied","Data":"05f52ff477dabfc65f4097a46befbf384c0cbb1c34c3583a560996c828162fb4"} Jul 22 04:37:33.211681 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:33.211312 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" event={"ID":"7a18c167-5511-43ba-a3d8-0091015bfd7b","Type":"ContainerStarted","Data":"316f0f305394600fcaad05465f3de86ccd236c881bde7619218727c155e7b3d1"} Jul 22 04:37:34.217638 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:34.217603 2572 generic.go:356] "Generic (PLEG): container finished" podID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerID="e9caad662b72068da29a3ccad80d593c5f53b7de517debb0b63b69865a6494f8" exitCode=0 Jul 22 04:37:34.218073 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:34.217680 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" event={"ID":"7a18c167-5511-43ba-a3d8-0091015bfd7b","Type":"ContainerDied","Data":"e9caad662b72068da29a3ccad80d593c5f53b7de517debb0b63b69865a6494f8"} Jul 22 04:37:35.224525 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:35.224482 2572 generic.go:356] "Generic (PLEG): container finished" podID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerID="67541efaa29f55c8259136673911a2f1a80b9dc296d9a446fb6d6309aeae4ec5" exitCode=0 Jul 22 04:37:35.224892 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:35.224564 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" event={"ID":"7a18c167-5511-43ba-a3d8-0091015bfd7b","Type":"ContainerDied","Data":"67541efaa29f55c8259136673911a2f1a80b9dc296d9a446fb6d6309aeae4ec5"} Jul 22 04:37:36.358426 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.358363 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:36.422703 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.422670 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhsmt\" (UniqueName: \"kubernetes.io/projected/7a18c167-5511-43ba-a3d8-0091015bfd7b-kube-api-access-zhsmt\") pod \"7a18c167-5511-43ba-a3d8-0091015bfd7b\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " Jul 22 04:37:36.422703 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.422706 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-bundle\") pod \"7a18c167-5511-43ba-a3d8-0091015bfd7b\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " Jul 22 04:37:36.422922 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.422742 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-util\") pod \"7a18c167-5511-43ba-a3d8-0091015bfd7b\" (UID: \"7a18c167-5511-43ba-a3d8-0091015bfd7b\") " Jul 22 04:37:36.423464 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.423423 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-bundle" (OuterVolumeSpecName: "bundle") pod "7a18c167-5511-43ba-a3d8-0091015bfd7b" (UID: "7a18c167-5511-43ba-a3d8-0091015bfd7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:37:36.424673 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.424648 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a18c167-5511-43ba-a3d8-0091015bfd7b-kube-api-access-zhsmt" (OuterVolumeSpecName: "kube-api-access-zhsmt") pod "7a18c167-5511-43ba-a3d8-0091015bfd7b" (UID: "7a18c167-5511-43ba-a3d8-0091015bfd7b"). InnerVolumeSpecName "kube-api-access-zhsmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:37:36.427721 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.427698 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-util" (OuterVolumeSpecName: "util") pod "7a18c167-5511-43ba-a3d8-0091015bfd7b" (UID: "7a18c167-5511-43ba-a3d8-0091015bfd7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:37:36.523740 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.523714 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsmt\" (UniqueName: \"kubernetes.io/projected/7a18c167-5511-43ba-a3d8-0091015bfd7b-kube-api-access-zhsmt\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:36.523740 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.523741 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:36.523893 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:36.523750 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a18c167-5511-43ba-a3d8-0091015bfd7b-util\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:37:37.235648 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:37.235566 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" Jul 22 04:37:37.235787 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:37.235574 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmskm7" event={"ID":"7a18c167-5511-43ba-a3d8-0091015bfd7b","Type":"ContainerDied","Data":"316f0f305394600fcaad05465f3de86ccd236c881bde7619218727c155e7b3d1"} Jul 22 04:37:37.235787 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:37.235687 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="316f0f305394600fcaad05465f3de86ccd236c881bde7619218727c155e7b3d1" Jul 22 04:37:47.919848 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.919809 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k"] Jul 22 04:37:47.920223 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.920150 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerName="util" Jul 22 04:37:47.920223 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.920162 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerName="util" Jul 22 04:37:47.920223 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.920175 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerName="pull" Jul 22 04:37:47.920223 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.920180 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerName="pull" Jul 22 04:37:47.920223 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.920187 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerName="extract" Jul 22 04:37:47.920223 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.920193 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerName="extract" Jul 22 04:37:47.920427 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.920255 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a18c167-5511-43ba-a3d8-0091015bfd7b" containerName="extract" Jul 22 04:37:47.926788 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.926770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:47.928864 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.928845 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 04:37:47.929150 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.929129 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 04:37:47.929443 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.929424 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 04:37:47.929539 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.929446 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 04:37:47.929539 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.929508 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nzjg5\"" Jul 22 04:37:47.935904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:47.935882 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k"] Jul 22 04:37:48.019829 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.019789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0014fc51-7b57-47b0-9be1-62a7567fd2df-tmp\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.019981 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.019842 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0014fc51-7b57-47b0-9be1-62a7567fd2df-webhook-cert\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.019981 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.019897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0014fc51-7b57-47b0-9be1-62a7567fd2df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.019981 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.019922 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snw44\" (UniqueName: \"kubernetes.io/projected/0014fc51-7b57-47b0-9be1-62a7567fd2df-kube-api-access-snw44\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.019981 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.019978 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0014fc51-7b57-47b0-9be1-62a7567fd2df-manifests\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.121235 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.121199 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0014fc51-7b57-47b0-9be1-62a7567fd2df-manifests\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.121361 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.121260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0014fc51-7b57-47b0-9be1-62a7567fd2df-tmp\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.121361 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.121301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0014fc51-7b57-47b0-9be1-62a7567fd2df-webhook-cert\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.121361 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.121343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0014fc51-7b57-47b0-9be1-62a7567fd2df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.121502 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.121400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snw44\" (UniqueName: \"kubernetes.io/projected/0014fc51-7b57-47b0-9be1-62a7567fd2df-kube-api-access-snw44\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.121703 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.121682 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0014fc51-7b57-47b0-9be1-62a7567fd2df-manifests\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.123725 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.123693 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0014fc51-7b57-47b0-9be1-62a7567fd2df-tmp\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.123823 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.123730 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0014fc51-7b57-47b0-9be1-62a7567fd2df-webhook-cert\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.123823 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.123805 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0014fc51-7b57-47b0-9be1-62a7567fd2df-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.129108 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.129086 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snw44\" (UniqueName: \"kubernetes.io/projected/0014fc51-7b57-47b0-9be1-62a7567fd2df-kube-api-access-snw44\") pod \"opendatahub-operator-controller-manager-5567574457-b6l2k\" (UID: \"0014fc51-7b57-47b0-9be1-62a7567fd2df\") " pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.237847 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.237768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:48.404566 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:48.404542 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k"] Jul 22 04:37:48.406685 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:37:48.406659 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0014fc51_7b57_47b0_9be1_62a7567fd2df.slice/crio-080b2596c04f1108dfd3386c2c82f8a6f2db5d73abc9535e268107c932b7bc71 WatchSource:0}: Error finding container 080b2596c04f1108dfd3386c2c82f8a6f2db5d73abc9535e268107c932b7bc71: Status 404 returned error can't find the container with id 080b2596c04f1108dfd3386c2c82f8a6f2db5d73abc9535e268107c932b7bc71 Jul 22 04:37:49.295121 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:49.295073 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" event={"ID":"0014fc51-7b57-47b0-9be1-62a7567fd2df","Type":"ContainerStarted","Data":"080b2596c04f1108dfd3386c2c82f8a6f2db5d73abc9535e268107c932b7bc71"} Jul 22 04:37:52.317437 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:52.317396 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" event={"ID":"0014fc51-7b57-47b0-9be1-62a7567fd2df","Type":"ContainerStarted","Data":"932c2c038d959461016eaa222f16911f9804d84fd52f64255e0ac9e90f12e94b"} Jul 22 04:37:53.323909 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:53.323874 2572 generic.go:356] "Generic (PLEG): container finished" podID="0014fc51-7b57-47b0-9be1-62a7567fd2df" containerID="932c2c038d959461016eaa222f16911f9804d84fd52f64255e0ac9e90f12e94b" exitCode=0 Jul 22 04:37:53.324271 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:53.323937 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" event={"ID":"0014fc51-7b57-47b0-9be1-62a7567fd2df","Type":"ContainerDied","Data":"932c2c038d959461016eaa222f16911f9804d84fd52f64255e0ac9e90f12e94b"} Jul 22 04:37:54.330074 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:54.330042 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" event={"ID":"0014fc51-7b57-47b0-9be1-62a7567fd2df","Type":"ContainerStarted","Data":"de0fc26ac4e355b534fb24b6ab76dafcdf6397ab14490f0dd73dec8e6d5737dc"} Jul 22 04:37:54.330471 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:54.330130 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:37:54.367854 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:37:54.367804 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" podStartSLOduration=3.042946126 podStartE2EDuration="7.367788739s" totalImagesPullingTime="3.303868157s" totalInitContainerRuntime="1.020974456s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:37:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:37:54.365525405 +0000 UTC m=+286.084130540" watchObservedRunningTime="2026-07-22 04:37:54.367788739 +0000 UTC m=+286.086393874" Jul 22 04:38:05.338363 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.338330 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5567574457-b6l2k" Jul 22 04:38:05.426232 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.426201 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz"] Jul 22 04:38:05.429407 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.429363 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.431851 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.431830 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 04:38:05.431851 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.431837 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 04:38:05.432024 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.431849 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 04:38:05.432024 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.431849 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6z5fv\"" Jul 22 04:38:05.432024 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.432009 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 04:38:05.432235 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.432220 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 04:38:05.438397 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.438361 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz"] Jul 22 04:38:05.577896 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.577863 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-metrics-cert\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.577896 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.577897 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-manager-config\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.578139 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.578022 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-cert\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.578139 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.578062 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjff\" (UniqueName: \"kubernetes.io/projected/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-kube-api-access-kbjff\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.679035 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.678960 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-cert\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.679035 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.678996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjff\" (UniqueName: \"kubernetes.io/projected/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-kube-api-access-kbjff\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.679221 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.679088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-metrics-cert\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.679221 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.679137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-manager-config\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.679797 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.679774 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-manager-config\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.681570 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.681545 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-cert\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.681770 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.681752 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-metrics-cert\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.686298 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.686272 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjff\" (UniqueName: \"kubernetes.io/projected/47b273c9-cbe8-4e85-9fd9-1d70e952a8ad-kube-api-access-kbjff\") pod \"lws-controller-manager-6f987c9cb4-2fdzz\" (UID: \"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad\") " pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.740069 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.740048 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:05.874876 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:05.874712 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz"] Jul 22 04:38:05.877370 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:38:05.877342 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b273c9_cbe8_4e85_9fd9_1d70e952a8ad.slice/crio-64a4f63ecb9a0005a9e2dac8843213d04278dd0e9409ff6a1727d39762184632 WatchSource:0}: Error finding container 64a4f63ecb9a0005a9e2dac8843213d04278dd0e9409ff6a1727d39762184632: Status 404 returned error can't find the container with id 64a4f63ecb9a0005a9e2dac8843213d04278dd0e9409ff6a1727d39762184632 Jul 22 04:38:06.388409 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:06.388357 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" event={"ID":"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad","Type":"ContainerStarted","Data":"64a4f63ecb9a0005a9e2dac8843213d04278dd0e9409ff6a1727d39762184632"} Jul 22 04:38:08.399645 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:08.399609 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" event={"ID":"47b273c9-cbe8-4e85-9fd9-1d70e952a8ad","Type":"ContainerStarted","Data":"4a828ee91c1c46db6c320ee28d12bc47aeecbae2a6f7ad622fb3f476959fbeb6"} Jul 22 04:38:08.400034 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:08.399732 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:08.670586 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:08.670508 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:38:08.678226 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:08.678198 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:38:08.683007 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:08.682984 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 04:38:15.205852 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.205797 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" podStartSLOduration=8.745287125 podStartE2EDuration="10.205777695s" totalImagesPullingTime="1.46049057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:38:08.447875398 +0000 UTC m=+300.166480545" watchObservedRunningTime="2026-07-22 04:38:15.205777695 +0000 UTC m=+306.924382831" Jul 22 04:38:15.208409 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.207765 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls"] Jul 22 04:38:15.210952 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.210931 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.213493 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.213469 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 04:38:15.213493 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.213483 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 04:38:15.213683 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.213469 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 04:38:15.213683 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.213473 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 04:38:15.213773 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.213697 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-25qrg\"" Jul 22 04:38:15.225454 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.225431 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls"] Jul 22 04:38:15.358847 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.358814 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.359001 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.358869 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04765fdd-a889-49af-b3eb-2d0fa42230cd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.359001 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.358927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.359001 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.358968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.359001 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.358987 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.359155 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.359021 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.359155 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.359050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmldl\" (UniqueName: \"kubernetes.io/projected/04765fdd-a889-49af-b3eb-2d0fa42230cd-kube-api-access-fmldl\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.460014 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.459939 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04765fdd-a889-49af-b3eb-2d0fa42230cd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.460014 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.459978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.460014 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.459999 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.460262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.460025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.460262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.460067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.460262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.460102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmldl\" (UniqueName: \"kubernetes.io/projected/04765fdd-a889-49af-b3eb-2d0fa42230cd-kube-api-access-fmldl\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.460262 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.460242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.460745 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.460713 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.462410 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.462369 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04765fdd-a889-49af-b3eb-2d0fa42230cd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.462504 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.462481 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.462719 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.462706 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.462775 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.462759 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.467466 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.467444 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04765fdd-a889-49af-b3eb-2d0fa42230cd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.467546 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.467518 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmldl\" (UniqueName: \"kubernetes.io/projected/04765fdd-a889-49af-b3eb-2d0fa42230cd-kube-api-access-fmldl\") pod \"istiod-openshift-gateway-94bb8fbfd-4gkls\" (UID: \"04765fdd-a889-49af-b3eb-2d0fa42230cd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.519854 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.519831 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:15.652964 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.652940 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls"] Jul 22 04:38:15.655760 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:38:15.655735 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04765fdd_a889_49af_b3eb_2d0fa42230cd.slice/crio-67e2e9bc04570ed923e657db8528cb9fc4ce2eda6ca5a69a0906fb1362ac5545 WatchSource:0}: Error finding container 67e2e9bc04570ed923e657db8528cb9fc4ce2eda6ca5a69a0906fb1362ac5545: Status 404 returned error can't find the container with id 67e2e9bc04570ed923e657db8528cb9fc4ce2eda6ca5a69a0906fb1362ac5545 Jul 22 04:38:15.657806 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:15.657789 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 04:38:16.439937 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:16.439895 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" event={"ID":"04765fdd-a889-49af-b3eb-2d0fa42230cd","Type":"ContainerStarted","Data":"67e2e9bc04570ed923e657db8528cb9fc4ce2eda6ca5a69a0906fb1362ac5545"} Jul 22 04:38:17.795390 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:17.795336 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 04:38:17.795676 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:17.795422 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 22 04:38:18.456265 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:18.456215 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" event={"ID":"04765fdd-a889-49af-b3eb-2d0fa42230cd","Type":"ContainerStarted","Data":"682dccc374097eaa55c15d062441849d6fcadbf002da14e7c2ec5152c3feabe7"} Jul 22 04:38:18.456476 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:18.456434 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:38:18.458429 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:18.458386 2572 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-4gkls container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 04:38:18.458568 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:18.458434 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" podUID="04765fdd-a889-49af-b3eb-2d0fa42230cd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 04:38:18.511018 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:18.510963 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" podStartSLOduration=1.373783559 podStartE2EDuration="3.510948212s" totalImagesPullingTime="2.137164653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:38:18.503937151 +0000 UTC m=+310.222542285" watchObservedRunningTime="2026-07-22 04:38:18.510948212 +0000 UTC m=+310.229553348" Jul 22 04:38:19.407692 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:19.407660 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f987c9cb4-2fdzz" Jul 22 04:38:19.462386 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:38:19.462345 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-4gkls" Jul 22 04:39:23.760199 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.760123 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb"] Jul 22 04:39:23.762757 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.762735 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.765143 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.765102 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 04:39:23.765143 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.765118 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 04:39:23.765904 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.765886 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g8w6n\"" Jul 22 04:39:23.771285 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.771264 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb"] Jul 22 04:39:23.817984 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.817957 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.818136 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.818018 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.818136 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.818064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvz2\" (UniqueName: \"kubernetes.io/projected/c49ffe21-6dc2-4c4a-997f-5151252bc373-kube-api-access-pxvz2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.918668 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.918629 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.918822 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.918687 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.918822 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.918722 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvz2\" (UniqueName: \"kubernetes.io/projected/c49ffe21-6dc2-4c4a-997f-5151252bc373-kube-api-access-pxvz2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.919008 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.918978 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.919008 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.918997 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:23.925956 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:23.925933 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvz2\" (UniqueName: \"kubernetes.io/projected/c49ffe21-6dc2-4c4a-997f-5151252bc373-kube-api-access-pxvz2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:24.074155 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.074125 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:24.200563 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.200534 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb"] Jul 22 04:39:24.202496 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:39:24.202453 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49ffe21_6dc2_4c4a_997f_5151252bc373.slice/crio-2720aa02df2511f9cb053aca7d910560e00e3a2bc5bcccb78da12b3b253ada8e WatchSource:0}: Error finding container 2720aa02df2511f9cb053aca7d910560e00e3a2bc5bcccb78da12b3b253ada8e: Status 404 returned error can't find the container with id 2720aa02df2511f9cb053aca7d910560e00e3a2bc5bcccb78da12b3b253ada8e Jul 22 04:39:24.363533 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.363461 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6"] Jul 22 04:39:24.365744 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.365728 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.376615 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.376590 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6"] Jul 22 04:39:24.423223 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.423193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.423354 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.423237 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.423354 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.423268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8l6\" (UniqueName: \"kubernetes.io/projected/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-kube-api-access-zn8l6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.524406 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.524360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.524548 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.524412 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8l6\" (UniqueName: \"kubernetes.io/projected/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-kube-api-access-zn8l6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.524632 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.524610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.524790 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.524771 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.524898 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.524884 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.532308 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.532284 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8l6\" (UniqueName: \"kubernetes.io/projected/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-kube-api-access-zn8l6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.675429 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.675328 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:24.775636 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.775602 2572 generic.go:356] "Generic (PLEG): container finished" podID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerID="a486fa1cfd1ac6bba5642dfe253bba310777b0a9df3099751f9fd1e72d2e8097" exitCode=0 Jul 22 04:39:24.776042 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.775677 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" event={"ID":"c49ffe21-6dc2-4c4a-997f-5151252bc373","Type":"ContainerDied","Data":"a486fa1cfd1ac6bba5642dfe253bba310777b0a9df3099751f9fd1e72d2e8097"} Jul 22 04:39:24.776042 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.775734 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" event={"ID":"c49ffe21-6dc2-4c4a-997f-5151252bc373","Type":"ContainerStarted","Data":"2720aa02df2511f9cb053aca7d910560e00e3a2bc5bcccb78da12b3b253ada8e"} Jul 22 04:39:24.801853 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.801829 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6"] Jul 22 04:39:24.803523 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:39:24.803495 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d143919_7db5_45a8_91e9_04ae1d5ff7e7.slice/crio-dc482ace2d42492a6a78306ee7c56783fa88c4ec8a84baddd01ed69f7da73ef5 WatchSource:0}: Error finding container dc482ace2d42492a6a78306ee7c56783fa88c4ec8a84baddd01ed69f7da73ef5: Status 404 returned error can't find the container with id dc482ace2d42492a6a78306ee7c56783fa88c4ec8a84baddd01ed69f7da73ef5 Jul 22 04:39:24.965680 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.965654 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh"] Jul 22 04:39:24.967942 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.967923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:24.977112 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:24.977091 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh"] Jul 22 04:39:25.029214 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.029184 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.029366 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.029281 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hcl\" (UniqueName: \"kubernetes.io/projected/9d0bcbfb-e89d-4952-a244-3d684923ac79-kube-api-access-95hcl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.029366 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.029334 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.129859 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.129828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hcl\" (UniqueName: \"kubernetes.io/projected/9d0bcbfb-e89d-4952-a244-3d684923ac79-kube-api-access-95hcl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.129984 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.129868 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.129984 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.129888 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.130215 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.130194 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.130288 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.130229 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.138111 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.138086 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hcl\" (UniqueName: \"kubernetes.io/projected/9d0bcbfb-e89d-4952-a244-3d684923ac79-kube-api-access-95hcl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.277007 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.276978 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:25.369962 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.369919 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg"] Jul 22 04:39:25.373634 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.373605 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.378938 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.378866 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg"] Jul 22 04:39:25.422239 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.422215 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh"] Jul 22 04:39:25.423442 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:39:25.423407 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0bcbfb_e89d_4952_a244_3d684923ac79.slice/crio-3a6932ac47f95485c663fce437969551ab16767b05686c5fbaf6da838313130c WatchSource:0}: Error finding container 3a6932ac47f95485c663fce437969551ab16767b05686c5fbaf6da838313130c: Status 404 returned error can't find the container with id 3a6932ac47f95485c663fce437969551ab16767b05686c5fbaf6da838313130c Jul 22 04:39:25.432293 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.432267 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgdt\" (UniqueName: \"kubernetes.io/projected/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-kube-api-access-xkgdt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.432419 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.432354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.432477 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.432419 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.533926 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.533854 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.533926 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.533904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.534106 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.533982 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgdt\" (UniqueName: \"kubernetes.io/projected/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-kube-api-access-xkgdt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.534295 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.534266 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.534584 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.534558 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.542015 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.541987 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgdt\" (UniqueName: \"kubernetes.io/projected/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-kube-api-access-xkgdt\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.688026 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.687997 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:25.784519 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.784429 2572 generic.go:356] "Generic (PLEG): container finished" podID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerID="122dfb2554e633a0d5c0707590a7edfee3ed44433ab1b190569779bc6932bee1" exitCode=0 Jul 22 04:39:25.784908 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.784513 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" event={"ID":"c49ffe21-6dc2-4c4a-997f-5151252bc373","Type":"ContainerDied","Data":"122dfb2554e633a0d5c0707590a7edfee3ed44433ab1b190569779bc6932bee1"} Jul 22 04:39:25.786285 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.786246 2572 generic.go:356] "Generic (PLEG): container finished" podID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerID="8b70bc4c112122c1d3f9709d0c1d51d0b9d212931c482b11e093c65ba1e75e59" exitCode=0 Jul 22 04:39:25.786418 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.786400 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" event={"ID":"7d143919-7db5-45a8-91e9-04ae1d5ff7e7","Type":"ContainerDied","Data":"8b70bc4c112122c1d3f9709d0c1d51d0b9d212931c482b11e093c65ba1e75e59"} Jul 22 04:39:25.786484 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.786442 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" event={"ID":"7d143919-7db5-45a8-91e9-04ae1d5ff7e7","Type":"ContainerStarted","Data":"dc482ace2d42492a6a78306ee7c56783fa88c4ec8a84baddd01ed69f7da73ef5"} Jul 22 04:39:25.789110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.788548 2572 generic.go:356] "Generic (PLEG): container finished" podID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerID="3e10494783255c5ecae6cb4c3a7ae5c1dfe5cff37bbb6e5f0a12bd2baaedece2" exitCode=0 Jul 22 04:39:25.789217 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.789117 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" event={"ID":"9d0bcbfb-e89d-4952-a244-3d684923ac79","Type":"ContainerDied","Data":"3e10494783255c5ecae6cb4c3a7ae5c1dfe5cff37bbb6e5f0a12bd2baaedece2"} Jul 22 04:39:25.789217 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.789160 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" event={"ID":"9d0bcbfb-e89d-4952-a244-3d684923ac79","Type":"ContainerStarted","Data":"3a6932ac47f95485c663fce437969551ab16767b05686c5fbaf6da838313130c"} Jul 22 04:39:25.818881 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:25.818775 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg"] Jul 22 04:39:25.821310 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:39:25.821288 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07d86d0_7d97_4657_9d50_c7cd80fe16a9.slice/crio-3de1d81c78203713c0570ea0fe532b6bf666728264b3835a32a72a986740a486 WatchSource:0}: Error finding container 3de1d81c78203713c0570ea0fe532b6bf666728264b3835a32a72a986740a486: Status 404 returned error can't find the container with id 3de1d81c78203713c0570ea0fe532b6bf666728264b3835a32a72a986740a486 Jul 22 04:39:26.796780 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.796752 2572 generic.go:356] "Generic (PLEG): container finished" podID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerID="2302eee82b39577ab42c4a0f90a36e9c1f8c5f5dbc805e2d73fd6b17dc553062" exitCode=0 Jul 22 04:39:26.802419 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.802395 2572 generic.go:356] "Generic (PLEG): container finished" podID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerID="397adcad2eaf014b5932f346a6033217c6992efcd4e77723495de9f3f1765d3e" exitCode=0 Jul 22 04:39:26.804281 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.804258 2572 generic.go:356] "Generic (PLEG): container finished" podID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerID="a3f7df8891eb67dbdec0bb6ba4108b66364176d79c736312aa22219fa0bc2216" exitCode=0 Jul 22 04:39:26.804788 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.804765 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" event={"ID":"9d0bcbfb-e89d-4952-a244-3d684923ac79","Type":"ContainerDied","Data":"2302eee82b39577ab42c4a0f90a36e9c1f8c5f5dbc805e2d73fd6b17dc553062"} Jul 22 04:39:26.804875 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.804802 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" event={"ID":"c49ffe21-6dc2-4c4a-997f-5151252bc373","Type":"ContainerDied","Data":"397adcad2eaf014b5932f346a6033217c6992efcd4e77723495de9f3f1765d3e"} Jul 22 04:39:26.804875 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.804825 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" event={"ID":"7d143919-7db5-45a8-91e9-04ae1d5ff7e7","Type":"ContainerDied","Data":"a3f7df8891eb67dbdec0bb6ba4108b66364176d79c736312aa22219fa0bc2216"} Jul 22 04:39:26.806100 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.806082 2572 generic.go:356] "Generic (PLEG): container finished" podID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerID="9b6dc319dab69443386e41bf4a3bb8b76b10cf6c4f3a50890770c60a038bb4dc" exitCode=0 Jul 22 04:39:26.806182 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.806113 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" event={"ID":"f07d86d0-7d97-4657-9d50-c7cd80fe16a9","Type":"ContainerDied","Data":"9b6dc319dab69443386e41bf4a3bb8b76b10cf6c4f3a50890770c60a038bb4dc"} Jul 22 04:39:26.806182 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:26.806140 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" event={"ID":"f07d86d0-7d97-4657-9d50-c7cd80fe16a9","Type":"ContainerStarted","Data":"3de1d81c78203713c0570ea0fe532b6bf666728264b3835a32a72a986740a486"} Jul 22 04:39:27.813409 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:27.813357 2572 generic.go:356] "Generic (PLEG): container finished" podID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerID="cb73324c67bcbde89048f202d37edc7333a250a7ac1db066da898e19a995ac2d" exitCode=0 Jul 22 04:39:27.813409 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:27.813406 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" event={"ID":"7d143919-7db5-45a8-91e9-04ae1d5ff7e7","Type":"ContainerDied","Data":"cb73324c67bcbde89048f202d37edc7333a250a7ac1db066da898e19a995ac2d"} Jul 22 04:39:27.815124 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:27.815104 2572 generic.go:356] "Generic (PLEG): container finished" podID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerID="f7bb97570db91ff72d78a01764f1c89e68f6f168f254b165109cc96c962cca6c" exitCode=0 Jul 22 04:39:27.815227 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:27.815182 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" event={"ID":"f07d86d0-7d97-4657-9d50-c7cd80fe16a9","Type":"ContainerDied","Data":"f7bb97570db91ff72d78a01764f1c89e68f6f168f254b165109cc96c962cca6c"} Jul 22 04:39:27.817467 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:27.817444 2572 generic.go:356] "Generic (PLEG): container finished" podID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerID="486defb26ffd7a718c698281c3543e563bae358c359dd24354b1861c660821c4" exitCode=0 Jul 22 04:39:27.817556 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:27.817499 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" event={"ID":"9d0bcbfb-e89d-4952-a244-3d684923ac79","Type":"ContainerDied","Data":"486defb26ffd7a718c698281c3543e563bae358c359dd24354b1861c660821c4"} Jul 22 04:39:27.954024 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:27.953998 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:28.056449 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.056415 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-bundle\") pod \"c49ffe21-6dc2-4c4a-997f-5151252bc373\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " Jul 22 04:39:28.056604 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.056494 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxvz2\" (UniqueName: \"kubernetes.io/projected/c49ffe21-6dc2-4c4a-997f-5151252bc373-kube-api-access-pxvz2\") pod \"c49ffe21-6dc2-4c4a-997f-5151252bc373\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " Jul 22 04:39:28.056604 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.056534 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-util\") pod \"c49ffe21-6dc2-4c4a-997f-5151252bc373\" (UID: \"c49ffe21-6dc2-4c4a-997f-5151252bc373\") " Jul 22 04:39:28.064646 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.057689 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-bundle" (OuterVolumeSpecName: "bundle") pod "c49ffe21-6dc2-4c4a-997f-5151252bc373" (UID: "c49ffe21-6dc2-4c4a-997f-5151252bc373"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:39:28.065513 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.065481 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49ffe21-6dc2-4c4a-997f-5151252bc373-kube-api-access-pxvz2" (OuterVolumeSpecName: "kube-api-access-pxvz2") pod "c49ffe21-6dc2-4c4a-997f-5151252bc373" (UID: "c49ffe21-6dc2-4c4a-997f-5151252bc373"). InnerVolumeSpecName "kube-api-access-pxvz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:39:28.065845 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.065802 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-util" (OuterVolumeSpecName: "util") pod "c49ffe21-6dc2-4c4a-997f-5151252bc373" (UID: "c49ffe21-6dc2-4c4a-997f-5151252bc373"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:39:28.158222 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.158194 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxvz2\" (UniqueName: \"kubernetes.io/projected/c49ffe21-6dc2-4c4a-997f-5151252bc373-kube-api-access-pxvz2\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:28.158222 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.158221 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-util\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:28.158392 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.158232 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c49ffe21-6dc2-4c4a-997f-5151252bc373-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:28.823628 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.823602 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" Jul 22 04:39:28.823958 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.823635 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb" event={"ID":"c49ffe21-6dc2-4c4a-997f-5151252bc373","Type":"ContainerDied","Data":"2720aa02df2511f9cb053aca7d910560e00e3a2bc5bcccb78da12b3b253ada8e"} Jul 22 04:39:28.823958 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.823683 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2720aa02df2511f9cb053aca7d910560e00e3a2bc5bcccb78da12b3b253ada8e" Jul 22 04:39:28.825825 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.825803 2572 generic.go:356] "Generic (PLEG): container finished" podID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerID="903490b2ac9ab69b0c7f0bf6993ca6b3f722db1996e5b0fc677b493f474b50d0" exitCode=0 Jul 22 04:39:28.825948 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.825873 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" event={"ID":"f07d86d0-7d97-4657-9d50-c7cd80fe16a9","Type":"ContainerDied","Data":"903490b2ac9ab69b0c7f0bf6993ca6b3f722db1996e5b0fc677b493f474b50d0"} Jul 22 04:39:28.977066 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.977040 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:28.978969 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:28.978949 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:29.064253 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.064180 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn8l6\" (UniqueName: \"kubernetes.io/projected/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-kube-api-access-zn8l6\") pod \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " Jul 22 04:39:29.064253 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.064223 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-bundle\") pod \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " Jul 22 04:39:29.064454 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.064256 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95hcl\" (UniqueName: \"kubernetes.io/projected/9d0bcbfb-e89d-4952-a244-3d684923ac79-kube-api-access-95hcl\") pod \"9d0bcbfb-e89d-4952-a244-3d684923ac79\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " Jul 22 04:39:29.064454 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.064288 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-util\") pod \"9d0bcbfb-e89d-4952-a244-3d684923ac79\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " Jul 22 04:39:29.064454 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.064325 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-bundle\") pod \"9d0bcbfb-e89d-4952-a244-3d684923ac79\" (UID: \"9d0bcbfb-e89d-4952-a244-3d684923ac79\") " Jul 22 04:39:29.064454 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.064364 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-util\") pod \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\" (UID: \"7d143919-7db5-45a8-91e9-04ae1d5ff7e7\") " Jul 22 04:39:29.064825 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.064790 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-bundle" (OuterVolumeSpecName: "bundle") pod "7d143919-7db5-45a8-91e9-04ae1d5ff7e7" (UID: "7d143919-7db5-45a8-91e9-04ae1d5ff7e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:39:29.064953 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.064929 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-bundle" (OuterVolumeSpecName: "bundle") pod "9d0bcbfb-e89d-4952-a244-3d684923ac79" (UID: "9d0bcbfb-e89d-4952-a244-3d684923ac79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:39:29.066336 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.066311 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-kube-api-access-zn8l6" (OuterVolumeSpecName: "kube-api-access-zn8l6") pod "7d143919-7db5-45a8-91e9-04ae1d5ff7e7" (UID: "7d143919-7db5-45a8-91e9-04ae1d5ff7e7"). InnerVolumeSpecName "kube-api-access-zn8l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:39:29.066753 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.066737 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0bcbfb-e89d-4952-a244-3d684923ac79-kube-api-access-95hcl" (OuterVolumeSpecName: "kube-api-access-95hcl") pod "9d0bcbfb-e89d-4952-a244-3d684923ac79" (UID: "9d0bcbfb-e89d-4952-a244-3d684923ac79"). InnerVolumeSpecName "kube-api-access-95hcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:39:29.069948 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.069920 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-util" (OuterVolumeSpecName: "util") pod "7d143919-7db5-45a8-91e9-04ae1d5ff7e7" (UID: "7d143919-7db5-45a8-91e9-04ae1d5ff7e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:39:29.070127 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.070107 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-util" (OuterVolumeSpecName: "util") pod "9d0bcbfb-e89d-4952-a244-3d684923ac79" (UID: "9d0bcbfb-e89d-4952-a244-3d684923ac79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:39:29.165052 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.165020 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8l6\" (UniqueName: \"kubernetes.io/projected/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-kube-api-access-zn8l6\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:29.165052 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.165048 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:29.165220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.165060 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95hcl\" (UniqueName: \"kubernetes.io/projected/9d0bcbfb-e89d-4952-a244-3d684923ac79-kube-api-access-95hcl\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:29.165220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.165069 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-util\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:29.165220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.165077 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0bcbfb-e89d-4952-a244-3d684923ac79-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:29.165220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.165084 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d143919-7db5-45a8-91e9-04ae1d5ff7e7-util\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:29.832340 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.832304 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" Jul 22 04:39:29.832340 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.832333 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh" event={"ID":"9d0bcbfb-e89d-4952-a244-3d684923ac79","Type":"ContainerDied","Data":"3a6932ac47f95485c663fce437969551ab16767b05686c5fbaf6da838313130c"} Jul 22 04:39:29.832886 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.832397 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6932ac47f95485c663fce437969551ab16767b05686c5fbaf6da838313130c" Jul 22 04:39:29.834174 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.834146 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" event={"ID":"7d143919-7db5-45a8-91e9-04ae1d5ff7e7","Type":"ContainerDied","Data":"dc482ace2d42492a6a78306ee7c56783fa88c4ec8a84baddd01ed69f7da73ef5"} Jul 22 04:39:29.834292 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.834189 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc482ace2d42492a6a78306ee7c56783fa88c4ec8a84baddd01ed69f7da73ef5" Jul 22 04:39:29.834292 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.834157 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6" Jul 22 04:39:29.880154 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:39:29.880124 2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d143919_7db5_45a8_91e9_04ae1d5ff7e7.slice\": RecentStats: unable to find data in memory cache]" Jul 22 04:39:29.971110 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:29.971089 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:39:30.070139 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.070105 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-bundle\") pod \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " Jul 22 04:39:30.070295 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.070148 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkgdt\" (UniqueName: \"kubernetes.io/projected/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-kube-api-access-xkgdt\") pod \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " Jul 22 04:39:30.070295 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.070212 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-util\") pod \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\" (UID: \"f07d86d0-7d97-4657-9d50-c7cd80fe16a9\") " Jul 22 04:39:30.070692 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.070660 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-bundle" (OuterVolumeSpecName: "bundle") pod "f07d86d0-7d97-4657-9d50-c7cd80fe16a9" (UID: "f07d86d0-7d97-4657-9d50-c7cd80fe16a9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:39:30.072339 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.072307 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-kube-api-access-xkgdt" (OuterVolumeSpecName: "kube-api-access-xkgdt") pod "f07d86d0-7d97-4657-9d50-c7cd80fe16a9" (UID: "f07d86d0-7d97-4657-9d50-c7cd80fe16a9"). InnerVolumeSpecName "kube-api-access-xkgdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:39:30.075534 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.075509 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-util" (OuterVolumeSpecName: "util") pod "f07d86d0-7d97-4657-9d50-c7cd80fe16a9" (UID: "f07d86d0-7d97-4657-9d50-c7cd80fe16a9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:39:30.171417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.171333 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:30.171417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.171391 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkgdt\" (UniqueName: \"kubernetes.io/projected/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-kube-api-access-xkgdt\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:30.171417 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.171403 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07d86d0-7d97-4657-9d50-c7cd80fe16a9-util\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:39:30.840997 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.840962 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" event={"ID":"f07d86d0-7d97-4657-9d50-c7cd80fe16a9","Type":"ContainerDied","Data":"3de1d81c78203713c0570ea0fe532b6bf666728264b3835a32a72a986740a486"} Jul 22 04:39:30.841327 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.841014 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3de1d81c78203713c0570ea0fe532b6bf666728264b3835a32a72a986740a486" Jul 22 04:39:30.841327 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:39:30.840970 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg" Jul 22 04:40:06.246746 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.246713 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j"] Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247049 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerName="util" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247060 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerName="util" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247069 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerName="extract" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247076 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerName="extract" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247085 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerName="extract" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247091 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerName="extract" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247100 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerName="extract" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247105 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerName="extract" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247112 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerName="pull" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247117 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerName="pull" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247123 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerName="util" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247128 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerName="util" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247135 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerName="extract" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247140 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerName="extract" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247149 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerName="util" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247154 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerName="util" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247163 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerName="pull" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247168 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerName="pull" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247177 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerName="pull" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247181 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerName="pull" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247188 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerName="util" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247193 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerName="util" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247200 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerName="pull" Jul 22 04:40:06.247208 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247205 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerName="pull" Jul 22 04:40:06.248033 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247260 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0bcbfb-e89d-4952-a244-3d684923ac79" containerName="extract" Jul 22 04:40:06.248033 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247270 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c49ffe21-6dc2-4c4a-997f-5151252bc373" containerName="extract" Jul 22 04:40:06.248033 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247277 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f07d86d0-7d97-4657-9d50-c7cd80fe16a9" containerName="extract" Jul 22 04:40:06.248033 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.247284 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d143919-7db5-45a8-91e9-04ae1d5ff7e7" containerName="extract" Jul 22 04:40:06.250675 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.250654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:06.253206 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.253185 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v4dp9\"" Jul 22 04:40:06.253563 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.253546 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 04:40:06.253806 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.253787 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 04:40:06.261872 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.261851 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j"] Jul 22 04:40:06.378304 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.378273 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5bxl\" (UniqueName: \"kubernetes.io/projected/785d6bff-feb9-496f-88f9-b07a707e5fd5-kube-api-access-x5bxl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmh6j\" (UID: \"785d6bff-feb9-496f-88f9-b07a707e5fd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:06.378567 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.378364 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/785d6bff-feb9-496f-88f9-b07a707e5fd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmh6j\" (UID: \"785d6bff-feb9-496f-88f9-b07a707e5fd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:06.479189 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.479141 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/785d6bff-feb9-496f-88f9-b07a707e5fd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmh6j\" (UID: \"785d6bff-feb9-496f-88f9-b07a707e5fd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:06.479350 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.479256 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5bxl\" (UniqueName: \"kubernetes.io/projected/785d6bff-feb9-496f-88f9-b07a707e5fd5-kube-api-access-x5bxl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmh6j\" (UID: \"785d6bff-feb9-496f-88f9-b07a707e5fd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:06.479642 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.479617 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/785d6bff-feb9-496f-88f9-b07a707e5fd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmh6j\" (UID: \"785d6bff-feb9-496f-88f9-b07a707e5fd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:06.486819 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.486799 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5bxl\" (UniqueName: \"kubernetes.io/projected/785d6bff-feb9-496f-88f9-b07a707e5fd5-kube-api-access-x5bxl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xmh6j\" (UID: \"785d6bff-feb9-496f-88f9-b07a707e5fd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:06.560881 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.560850 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:06.695939 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.695910 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j"] Jul 22 04:40:06.697963 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:40:06.697934 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785d6bff_feb9_496f_88f9_b07a707e5fd5.slice/crio-6d29e42f9c8060e2878da999396b42966ca3ab76fa49d3220dc5151cf9bbdca2 WatchSource:0}: Error finding container 6d29e42f9c8060e2878da999396b42966ca3ab76fa49d3220dc5151cf9bbdca2: Status 404 returned error can't find the container with id 6d29e42f9c8060e2878da999396b42966ca3ab76fa49d3220dc5151cf9bbdca2 Jul 22 04:40:06.866221 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.866128 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j"] Jul 22 04:40:06.875105 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.875071 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j"] Jul 22 04:40:06.921869 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.921837 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p"] Jul 22 04:40:06.926552 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.926522 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:06.956090 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:06.954596 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p"] Jul 22 04:40:07.084199 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:07.084164 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbwlv\" (UniqueName: \"kubernetes.io/projected/2bd93c33-d053-4b61-b164-3931babef813-kube-api-access-pbwlv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t6m4p\" (UID: \"2bd93c33-d053-4b61-b164-3931babef813\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:07.084199 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:07.084204 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bd93c33-d053-4b61-b164-3931babef813-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t6m4p\" (UID: \"2bd93c33-d053-4b61-b164-3931babef813\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:07.185515 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:07.185437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwlv\" (UniqueName: \"kubernetes.io/projected/2bd93c33-d053-4b61-b164-3931babef813-kube-api-access-pbwlv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t6m4p\" (UID: \"2bd93c33-d053-4b61-b164-3931babef813\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:07.185515 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:07.185476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bd93c33-d053-4b61-b164-3931babef813-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t6m4p\" (UID: \"2bd93c33-d053-4b61-b164-3931babef813\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:07.185790 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:07.185772 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bd93c33-d053-4b61-b164-3931babef813-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t6m4p\" (UID: \"2bd93c33-d053-4b61-b164-3931babef813\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:07.200362 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:07.200332 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwlv\" (UniqueName: \"kubernetes.io/projected/2bd93c33-d053-4b61-b164-3931babef813-kube-api-access-pbwlv\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-t6m4p\" (UID: \"2bd93c33-d053-4b61-b164-3931babef813\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:07.238757 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:07.238716 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:07.570912 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:07.570889 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p"] Jul 22 04:40:07.573258 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:40:07.573229 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd93c33_d053_4b61_b164_3931babef813.slice/crio-ec64ade09af5e7b8d4d1d65b6ce32db557b750b2cf0866a3876d43fccfd10279 WatchSource:0}: Error finding container ec64ade09af5e7b8d4d1d65b6ce32db557b750b2cf0866a3876d43fccfd10279: Status 404 returned error can't find the container with id ec64ade09af5e7b8d4d1d65b6ce32db557b750b2cf0866a3876d43fccfd10279 Jul 22 04:40:08.027175 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:08.027143 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" event={"ID":"2bd93c33-d053-4b61-b164-3931babef813","Type":"ContainerStarted","Data":"ec64ade09af5e7b8d4d1d65b6ce32db557b750b2cf0866a3876d43fccfd10279"} Jul 22 04:40:12.051712 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.051668 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" podUID="785d6bff-feb9-496f-88f9-b07a707e5fd5" containerName="manager" containerID="cri-o://cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b" gracePeriod=2 Jul 22 04:40:12.053285 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.053257 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" event={"ID":"2bd93c33-d053-4b61-b164-3931babef813","Type":"ContainerStarted","Data":"e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a"} Jul 22 04:40:12.053443 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.053310 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:12.079258 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.079194 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" podStartSLOduration=2.117022342 podStartE2EDuration="6.079172837s" totalImagesPullingTime="3.962150495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:40:12.075960019 +0000 UTC m=+423.794565155" watchObservedRunningTime="2026-07-22 04:40:12.079172837 +0000 UTC m=+423.797777973" Jul 22 04:40:12.291317 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.291291 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:12.293459 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:40:12.293433 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xmh6j\" is forbidden: User \"system:node:ip-10-0-142-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-47.ec2.internal' and this object" podUID="785d6bff-feb9-496f-88f9-b07a707e5fd5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:12.335952 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.335737 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/785d6bff-feb9-496f-88f9-b07a707e5fd5-extensions-socket-volume\") pod \"785d6bff-feb9-496f-88f9-b07a707e5fd5\" (UID: \"785d6bff-feb9-496f-88f9-b07a707e5fd5\") " Jul 22 04:40:12.336108 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.335999 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5bxl\" (UniqueName: \"kubernetes.io/projected/785d6bff-feb9-496f-88f9-b07a707e5fd5-kube-api-access-x5bxl\") pod \"785d6bff-feb9-496f-88f9-b07a707e5fd5\" (UID: \"785d6bff-feb9-496f-88f9-b07a707e5fd5\") " Jul 22 04:40:12.336312 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.336278 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785d6bff-feb9-496f-88f9-b07a707e5fd5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "785d6bff-feb9-496f-88f9-b07a707e5fd5" (UID: "785d6bff-feb9-496f-88f9-b07a707e5fd5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:40:12.343924 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.343893 2572 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/785d6bff-feb9-496f-88f9-b07a707e5fd5-extensions-socket-volume\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:12.344931 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.344907 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785d6bff-feb9-496f-88f9-b07a707e5fd5-kube-api-access-x5bxl" (OuterVolumeSpecName: "kube-api-access-x5bxl") pod "785d6bff-feb9-496f-88f9-b07a707e5fd5" (UID: "785d6bff-feb9-496f-88f9-b07a707e5fd5"). InnerVolumeSpecName "kube-api-access-x5bxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:40:12.444514 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.444475 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5bxl\" (UniqueName: \"kubernetes.io/projected/785d6bff-feb9-496f-88f9-b07a707e5fd5-kube-api-access-x5bxl\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:12.801543 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:12.801510 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="785d6bff-feb9-496f-88f9-b07a707e5fd5" path="/var/lib/kubelet/pods/785d6bff-feb9-496f-88f9-b07a707e5fd5/volumes" Jul 22 04:40:13.060095 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:13.059999 2572 generic.go:356] "Generic (PLEG): container finished" podID="785d6bff-feb9-496f-88f9-b07a707e5fd5" containerID="cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b" exitCode=2 Jul 22 04:40:13.060095 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:13.060053 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:13.060605 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:13.060101 2572 scope.go:117] "RemoveContainer" containerID="cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b" Jul 22 04:40:13.064473 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:40:13.064443 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xmh6j\" is forbidden: User \"system:node:ip-10-0-142-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-47.ec2.internal' and this object" podUID="785d6bff-feb9-496f-88f9-b07a707e5fd5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xmh6j" Jul 22 04:40:13.069166 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:13.069150 2572 scope.go:117] "RemoveContainer" containerID="cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b" Jul 22 04:40:13.069466 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:40:13.069445 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b\": container with ID starting with cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b not found: ID does not exist" containerID="cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b" Jul 22 04:40:13.069527 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:13.069474 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b"} err="failed to get container status \"cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b\": rpc error: code = NotFound desc = could not find container \"cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b\": container with ID starting with cfaf4fd9a1d7093b91e39e01565e35600cf82951eff448505b0b7d43be6af29b not found: ID does not exist" Jul 22 04:40:23.062715 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:23.062681 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:25.453490 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:25.453458 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c97c776db-2grgr"] Jul 22 04:40:35.446854 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.446733 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p"] Jul 22 04:40:35.447271 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.446975 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" podUID="2bd93c33-d053-4b61-b164-3931babef813" containerName="manager" containerID="cri-o://e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a" gracePeriod=10 Jul 22 04:40:35.696610 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.696588 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:35.846179 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.846151 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbwlv\" (UniqueName: \"kubernetes.io/projected/2bd93c33-d053-4b61-b164-3931babef813-kube-api-access-pbwlv\") pod \"2bd93c33-d053-4b61-b164-3931babef813\" (UID: \"2bd93c33-d053-4b61-b164-3931babef813\") " Jul 22 04:40:35.846332 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.846214 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bd93c33-d053-4b61-b164-3931babef813-extensions-socket-volume\") pod \"2bd93c33-d053-4b61-b164-3931babef813\" (UID: \"2bd93c33-d053-4b61-b164-3931babef813\") " Jul 22 04:40:35.846663 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.846628 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd93c33-d053-4b61-b164-3931babef813-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2bd93c33-d053-4b61-b164-3931babef813" (UID: "2bd93c33-d053-4b61-b164-3931babef813"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 04:40:35.848068 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.848038 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd93c33-d053-4b61-b164-3931babef813-kube-api-access-pbwlv" (OuterVolumeSpecName: "kube-api-access-pbwlv") pod "2bd93c33-d053-4b61-b164-3931babef813" (UID: "2bd93c33-d053-4b61-b164-3931babef813"). InnerVolumeSpecName "kube-api-access-pbwlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:40:35.947245 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.947216 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbwlv\" (UniqueName: \"kubernetes.io/projected/2bd93c33-d053-4b61-b164-3931babef813-kube-api-access-pbwlv\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:35.947245 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:35.947247 2572 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2bd93c33-d053-4b61-b164-3931babef813-extensions-socket-volume\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:36.181967 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.181879 2572 generic.go:356] "Generic (PLEG): container finished" podID="2bd93c33-d053-4b61-b164-3931babef813" containerID="e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a" exitCode=0 Jul 22 04:40:36.181967 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.181925 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" Jul 22 04:40:36.181967 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.181940 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" event={"ID":"2bd93c33-d053-4b61-b164-3931babef813","Type":"ContainerDied","Data":"e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a"} Jul 22 04:40:36.182177 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.181991 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p" event={"ID":"2bd93c33-d053-4b61-b164-3931babef813","Type":"ContainerDied","Data":"ec64ade09af5e7b8d4d1d65b6ce32db557b750b2cf0866a3876d43fccfd10279"} Jul 22 04:40:36.182177 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.182023 2572 scope.go:117] "RemoveContainer" containerID="e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a" Jul 22 04:40:36.193004 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.192985 2572 scope.go:117] "RemoveContainer" containerID="e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a" Jul 22 04:40:36.193447 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:40:36.193421 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a\": container with ID starting with e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a not found: ID does not exist" containerID="e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a" Jul 22 04:40:36.193532 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.193456 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a"} err="failed to get container status \"e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a\": rpc error: code = NotFound desc = could not find container \"e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a\": container with ID starting with e43950e8991d936ed68af7fb191a980e3e4d91db2db256d8f2bcd1b0e1daf31a not found: ID does not exist" Jul 22 04:40:36.209701 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.209674 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p"] Jul 22 04:40:36.217794 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.217770 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-t6m4p"] Jul 22 04:40:36.801155 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:36.801120 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bd93c33-d053-4b61-b164-3931babef813" path="/var/lib/kubelet/pods/2bd93c33-d053-4b61-b164-3931babef813/volumes" Jul 22 04:40:50.475060 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.474998 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c97c776db-2grgr" podUID="ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" containerName="console" containerID="cri-o://7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a" gracePeriod=15 Jul 22 04:40:50.730035 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.729980 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:40:50.870280 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870243 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-config\") pod \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " Jul 22 04:40:50.870459 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870308 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-trusted-ca-bundle\") pod \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " Jul 22 04:40:50.870459 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870429 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdrgt\" (UniqueName: \"kubernetes.io/projected/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-kube-api-access-qdrgt\") pod \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " Jul 22 04:40:50.870590 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870496 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-serving-cert\") pod \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " Jul 22 04:40:50.870590 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870550 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-oauth-serving-cert\") pod \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " Jul 22 04:40:50.870699 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870629 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-oauth-config\") pod \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " Jul 22 04:40:50.870699 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870660 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-service-ca\") pod \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\" (UID: \"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f\") " Jul 22 04:40:50.870800 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870687 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-config" (OuterVolumeSpecName: "console-config") pod "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" (UID: "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:40:50.870800 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.870785 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" (UID: "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:40:50.871044 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.871025 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-config\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:50.871202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.871182 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-trusted-ca-bundle\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:50.871202 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.871037 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" (UID: "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:40:50.871339 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.871113 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-service-ca" (OuterVolumeSpecName: "service-ca") pod "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" (UID: "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 04:40:50.872670 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.872649 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-kube-api-access-qdrgt" (OuterVolumeSpecName: "kube-api-access-qdrgt") pod "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" (UID: "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f"). InnerVolumeSpecName "kube-api-access-qdrgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 04:40:50.872763 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.872690 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" (UID: "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:40:50.872819 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.872765 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" (UID: "ddc82bd7-3e98-4cf8-bc7f-373a4709f95f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 04:40:50.971827 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.971798 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdrgt\" (UniqueName: \"kubernetes.io/projected/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-kube-api-access-qdrgt\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:50.971827 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.971827 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-serving-cert\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:50.972016 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.971841 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-oauth-serving-cert\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:50.972016 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.971854 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-console-oauth-config\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:50.972016 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:50.971870 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 04:40:51.261532 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.261496 2572 generic.go:356] "Generic (PLEG): container finished" podID="ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" containerID="7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a" exitCode=0 Jul 22 04:40:51.261674 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.261565 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97c776db-2grgr" event={"ID":"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f","Type":"ContainerDied","Data":"7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a"} Jul 22 04:40:51.261674 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.261607 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c97c776db-2grgr" Jul 22 04:40:51.261674 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.261628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c97c776db-2grgr" event={"ID":"ddc82bd7-3e98-4cf8-bc7f-373a4709f95f","Type":"ContainerDied","Data":"a8b1b5b423ef0706bc08b261b4a88144ddc87b6063de15379bd937952389d1de"} Jul 22 04:40:51.261674 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.261654 2572 scope.go:117] "RemoveContainer" containerID="7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a" Jul 22 04:40:51.271581 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.271562 2572 scope.go:117] "RemoveContainer" containerID="7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a" Jul 22 04:40:51.271846 ip-10-0-142-47 kubenswrapper[2572]: E0722 04:40:51.271825 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a\": container with ID starting with 7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a not found: ID does not exist" containerID="7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a" Jul 22 04:40:51.271960 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.271854 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a"} err="failed to get container status \"7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a\": rpc error: code = NotFound desc = could not find container \"7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a\": container with ID starting with 7f218eb68c0be7a065bd7716e7aa94395b5ec7e31c871c4770a1b30531fba77a not found: ID does not exist" Jul 22 04:40:51.283043 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.283019 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c97c776db-2grgr"] Jul 22 04:40:51.286738 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.286718 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c97c776db-2grgr"] Jul 22 04:40:51.927226 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927189 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:40:51.927643 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927531 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="785d6bff-feb9-496f-88f9-b07a707e5fd5" containerName="manager" Jul 22 04:40:51.927643 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927542 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="785d6bff-feb9-496f-88f9-b07a707e5fd5" containerName="manager" Jul 22 04:40:51.927643 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927555 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd93c33-d053-4b61-b164-3931babef813" containerName="manager" Jul 22 04:40:51.927643 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927561 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd93c33-d053-4b61-b164-3931babef813" containerName="manager" Jul 22 04:40:51.927643 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927575 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" containerName="console" Jul 22 04:40:51.927643 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927580 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" containerName="console" Jul 22 04:40:51.927882 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927654 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="785d6bff-feb9-496f-88f9-b07a707e5fd5" containerName="manager" Jul 22 04:40:51.927882 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927663 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd93c33-d053-4b61-b164-3931babef813" containerName="manager" Jul 22 04:40:51.927882 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.927671 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" containerName="console" Jul 22 04:40:51.930158 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.930141 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:40:51.932393 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.932355 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 04:40:51.932531 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.932413 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g8w6n\"" Jul 22 04:40:51.932531 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.932464 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 04:40:51.933288 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.933272 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 04:40:51.939808 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.939786 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:40:51.970188 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:51.970159 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:40:52.082694 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.082657 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2fe640d5-0ec1-475d-bb87-5d0deaa29ed9-config-file\") pod \"limitador-limitador-54d6d988d5-4msqs\" (UID: \"2fe640d5-0ec1-475d-bb87-5d0deaa29ed9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:40:52.082865 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.082699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdjp\" (UniqueName: \"kubernetes.io/projected/2fe640d5-0ec1-475d-bb87-5d0deaa29ed9-kube-api-access-vzdjp\") pod \"limitador-limitador-54d6d988d5-4msqs\" (UID: \"2fe640d5-0ec1-475d-bb87-5d0deaa29ed9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:40:52.183774 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.183695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdjp\" (UniqueName: \"kubernetes.io/projected/2fe640d5-0ec1-475d-bb87-5d0deaa29ed9-kube-api-access-vzdjp\") pod \"limitador-limitador-54d6d988d5-4msqs\" (UID: \"2fe640d5-0ec1-475d-bb87-5d0deaa29ed9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:40:52.183906 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.183797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2fe640d5-0ec1-475d-bb87-5d0deaa29ed9-config-file\") pod \"limitador-limitador-54d6d988d5-4msqs\" (UID: \"2fe640d5-0ec1-475d-bb87-5d0deaa29ed9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:40:52.184406 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.184371 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2fe640d5-0ec1-475d-bb87-5d0deaa29ed9-config-file\") pod \"limitador-limitador-54d6d988d5-4msqs\" (UID: \"2fe640d5-0ec1-475d-bb87-5d0deaa29ed9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:40:52.191595 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.191568 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdjp\" (UniqueName: \"kubernetes.io/projected/2fe640d5-0ec1-475d-bb87-5d0deaa29ed9-kube-api-access-vzdjp\") pod \"limitador-limitador-54d6d988d5-4msqs\" (UID: \"2fe640d5-0ec1-475d-bb87-5d0deaa29ed9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:40:52.240538 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.240501 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:40:52.373797 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.373770 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:40:52.375135 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:40:52.375105 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe640d5_0ec1_475d_bb87_5d0deaa29ed9.slice/crio-904bf3c19509054558377dd4db5f54e497f4bc86bd8ba34382e487dae1bba359 WatchSource:0}: Error finding container 904bf3c19509054558377dd4db5f54e497f4bc86bd8ba34382e487dae1bba359: Status 404 returned error can't find the container with id 904bf3c19509054558377dd4db5f54e497f4bc86bd8ba34382e487dae1bba359 Jul 22 04:40:52.802638 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:52.802607 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc82bd7-3e98-4cf8-bc7f-373a4709f95f" path="/var/lib/kubelet/pods/ddc82bd7-3e98-4cf8-bc7f-373a4709f95f/volumes" Jul 22 04:40:53.278273 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:53.278237 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" event={"ID":"2fe640d5-0ec1-475d-bb87-5d0deaa29ed9","Type":"ContainerStarted","Data":"904bf3c19509054558377dd4db5f54e497f4bc86bd8ba34382e487dae1bba359"} Jul 22 04:40:55.291689 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:55.291651 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" event={"ID":"2fe640d5-0ec1-475d-bb87-5d0deaa29ed9","Type":"ContainerStarted","Data":"4a738b6508c9626cb85245229cbd198b6f1e9f30fcbe0e6577793eac33f8cc11"} Jul 22 04:40:55.292055 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:40:55.291724 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:41:06.298075 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:41:06.298039 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" Jul 22 04:41:06.313491 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:41:06.313298 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-4msqs" podStartSLOduration=12.793756337 podStartE2EDuration="15.313278244s" totalImagesPullingTime="2.519521907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:40:55.306337656 +0000 UTC m=+467.024942792" watchObservedRunningTime="2026-07-22 04:41:06.313278244 +0000 UTC m=+478.031883380" Jul 22 04:42:03.244168 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.244094 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vgrwr"] Jul 22 04:42:03.247470 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.247451 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:03.249818 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.249795 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ghjwv\"" Jul 22 04:42:03.249927 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.249870 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 04:42:03.257493 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.257471 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vgrwr"] Jul 22 04:42:03.341849 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.341819 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:42:03.352601 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.352558 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1218d7b6-1165-4ae1-a285-a85ea651eb82-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vgrwr\" (UID: \"1218d7b6-1165-4ae1-a285-a85ea651eb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:03.352721 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.352629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzl2p\" (UniqueName: \"kubernetes.io/projected/1218d7b6-1165-4ae1-a285-a85ea651eb82-kube-api-access-dzl2p\") pod \"payload-processing-5bf47cd988-vgrwr\" (UID: \"1218d7b6-1165-4ae1-a285-a85ea651eb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:03.454036 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.454005 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1218d7b6-1165-4ae1-a285-a85ea651eb82-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vgrwr\" (UID: \"1218d7b6-1165-4ae1-a285-a85ea651eb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:03.454220 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.454064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzl2p\" (UniqueName: \"kubernetes.io/projected/1218d7b6-1165-4ae1-a285-a85ea651eb82-kube-api-access-dzl2p\") pod \"payload-processing-5bf47cd988-vgrwr\" (UID: \"1218d7b6-1165-4ae1-a285-a85ea651eb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:03.454775 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.454747 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1218d7b6-1165-4ae1-a285-a85ea651eb82-plugins-config-volume\") pod \"payload-processing-5bf47cd988-vgrwr\" (UID: \"1218d7b6-1165-4ae1-a285-a85ea651eb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:03.462234 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.462211 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzl2p\" (UniqueName: \"kubernetes.io/projected/1218d7b6-1165-4ae1-a285-a85ea651eb82-kube-api-access-dzl2p\") pod \"payload-processing-5bf47cd988-vgrwr\" (UID: \"1218d7b6-1165-4ae1-a285-a85ea651eb82\") " pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:03.558337 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.558310 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:03.682641 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:03.682607 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-vgrwr"] Jul 22 04:42:03.684212 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:42:03.684183 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1218d7b6_1165_4ae1_a285_a85ea651eb82.slice/crio-5b0c438d9eaa897eeae3dad74cd9681f2116102cc974b623131b1dd22c49d2d8 WatchSource:0}: Error finding container 5b0c438d9eaa897eeae3dad74cd9681f2116102cc974b623131b1dd22c49d2d8: Status 404 returned error can't find the container with id 5b0c438d9eaa897eeae3dad74cd9681f2116102cc974b623131b1dd22c49d2d8 Jul 22 04:42:04.640070 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:04.640030 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" event={"ID":"1218d7b6-1165-4ae1-a285-a85ea651eb82","Type":"ContainerStarted","Data":"5b0c438d9eaa897eeae3dad74cd9681f2116102cc974b623131b1dd22c49d2d8"} Jul 22 04:42:05.646596 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:05.646564 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" event={"ID":"1218d7b6-1165-4ae1-a285-a85ea651eb82","Type":"ContainerStarted","Data":"7a122f7356997cdfe75ec8ec99ab610ec3e838437b965c1b2c2f04ac3d98d4c6"} Jul 22 04:42:05.647054 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:05.646621 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:05.661593 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:05.661540 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" podStartSLOduration=1.130070037 podStartE2EDuration="2.661525808s" totalImagesPullingTime="1.531455771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:42:05.660212527 +0000 UTC m=+537.378817666" watchObservedRunningTime="2026-07-22 04:42:05.661525808 +0000 UTC m=+537.380130943" Jul 22 04:42:16.654217 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:16.654184 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-vgrwr" Jul 22 04:42:23.374126 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:23.374089 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:42:45.159473 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:45.159443 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:42:55.061769 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:42:55.061732 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:43:04.457124 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.457091 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss"] Jul 22 04:43:04.460480 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.460463 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.465017 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.464989 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 04:43:04.465017 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.464997 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6ntgm\"" Jul 22 04:43:04.465180 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.464990 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 22 04:43:04.465180 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.465048 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 04:43:04.470439 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.470418 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss"] Jul 22 04:43:04.571530 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.571500 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.571710 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.571538 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.571710 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.571561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0adfbec2-eb35-433c-836b-2e9ca3ed7752-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.571710 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.571590 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.571710 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.571628 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lns8p\" (UniqueName: \"kubernetes.io/projected/0adfbec2-eb35-433c-836b-2e9ca3ed7752-kube-api-access-lns8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.571710 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.571655 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672097 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672097 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672103 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672335 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672128 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0adfbec2-eb35-433c-836b-2e9ca3ed7752-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672335 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672253 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672335 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lns8p\" (UniqueName: \"kubernetes.io/projected/0adfbec2-eb35-433c-836b-2e9ca3ed7752-kube-api-access-lns8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672575 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672635 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672569 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672635 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672615 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.672746 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.672728 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.674469 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.674444 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0adfbec2-eb35-433c-836b-2e9ca3ed7752-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.674684 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.674664 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0adfbec2-eb35-433c-836b-2e9ca3ed7752-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.679994 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.679972 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lns8p\" (UniqueName: \"kubernetes.io/projected/0adfbec2-eb35-433c-836b-2e9ca3ed7752-kube-api-access-lns8p\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss\" (UID: \"0adfbec2-eb35-433c-836b-2e9ca3ed7752\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.771803 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.771772 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:04.899311 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.899289 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss"] Jul 22 04:43:04.901250 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:43:04.901224 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adfbec2_eb35_433c_836b_2e9ca3ed7752.slice/crio-e8f8382d7ee26c12e80659d3a07a202179ed0e2e832e423deb9acc58f3cbe407 WatchSource:0}: Error finding container e8f8382d7ee26c12e80659d3a07a202179ed0e2e832e423deb9acc58f3cbe407: Status 404 returned error can't find the container with id e8f8382d7ee26c12e80659d3a07a202179ed0e2e832e423deb9acc58f3cbe407 Jul 22 04:43:04.937800 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:04.937776 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" event={"ID":"0adfbec2-eb35-433c-836b-2e9ca3ed7752","Type":"ContainerStarted","Data":"e8f8382d7ee26c12e80659d3a07a202179ed0e2e832e423deb9acc58f3cbe407"} Jul 22 04:43:06.351953 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:06.351914 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:43:06.949457 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:06.949414 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" event={"ID":"0adfbec2-eb35-433c-836b-2e9ca3ed7752","Type":"ContainerStarted","Data":"e9a878835462fc6eae01e74bffccfabde53b6c129d59a8e08dbff42b36aaf131"} Jul 22 04:43:06.967355 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:06.967301 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" podStartSLOduration=1.971981567 podStartE2EDuration="2.967286695s" totalImagesPullingTime="995.305128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 04:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 04:43:06.965512093 +0000 UTC m=+598.684117239" watchObservedRunningTime="2026-07-22 04:43:06.967286695 +0000 UTC m=+598.685891830" Jul 22 04:43:08.715230 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:08.715198 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:43:08.717050 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:08.717023 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:43:12.459286 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:12.459248 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:43:14.772759 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:14.772719 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:14.773165 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:14.772820 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:14.785256 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:14.785232 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:15.003423 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:15.003388 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss" Jul 22 04:43:28.766003 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:43:28.765968 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:45:00.142254 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:00.142220 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf"] Jul 22 04:45:00.145780 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:00.145756 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf" Jul 22 04:45:00.147977 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:00.147958 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 04:45:00.152087 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:00.152061 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf"] Jul 22 04:45:00.156790 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:00.156769 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf" Jul 22 04:45:00.283510 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:00.283482 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf"] Jul 22 04:45:00.285482 ip-10-0-142-47 kubenswrapper[2572]: W0722 04:45:00.285450 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb844ba8d_2d5c_4680_a624_154807fe27bf.slice/crio-614f4f7c42cc70f0207cfdc4408eed2da518d0588e7e7e4a62590cebe3edf8e9 WatchSource:0}: Error finding container 614f4f7c42cc70f0207cfdc4408eed2da518d0588e7e7e4a62590cebe3edf8e9: Status 404 returned error can't find the container with id 614f4f7c42cc70f0207cfdc4408eed2da518d0588e7e7e4a62590cebe3edf8e9 Jul 22 04:45:00.287635 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:00.287618 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 04:45:00.545994 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:00.545959 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf" event={"ID":"b844ba8d-2d5c-4680-a624-154807fe27bf","Type":"ContainerStarted","Data":"614f4f7c42cc70f0207cfdc4408eed2da518d0588e7e7e4a62590cebe3edf8e9"} Jul 22 04:45:03.564552 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:03.564509 2572 generic.go:356] "Generic (PLEG): container finished" podID="b844ba8d-2d5c-4680-a624-154807fe27bf" containerID="dd31791867ee36010787cd6e55dc5a8d138ec30ef18a5d88700d3ce7f86350d2" exitCode=0 Jul 22 04:45:03.564913 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:03.564592 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf" event={"ID":"b844ba8d-2d5c-4680-a624-154807fe27bf","Type":"ContainerDied","Data":"dd31791867ee36010787cd6e55dc5a8d138ec30ef18a5d88700d3ce7f86350d2"} Jul 22 04:45:04.698088 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:04.698064 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf" Jul 22 04:45:05.574774 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:05.574736 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf" event={"ID":"b844ba8d-2d5c-4680-a624-154807fe27bf","Type":"ContainerDied","Data":"614f4f7c42cc70f0207cfdc4408eed2da518d0588e7e7e4a62590cebe3edf8e9"} Jul 22 04:45:05.574774 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:05.574761 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744925-tspcf" Jul 22 04:45:05.575036 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:45:05.574785 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="614f4f7c42cc70f0207cfdc4408eed2da518d0588e7e7e4a62590cebe3edf8e9" Jul 22 04:46:07.955851 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:46:07.955810 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:46:17.257217 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:46:17.257183 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:46:28.265232 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:46:28.265196 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:48:08.751656 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:48:08.751576 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:48:08.752129 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:48:08.752098 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:48:12.958664 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:48:12.958629 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:48:21.965959 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:48:21.965921 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:48:48.656603 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:48:48.656567 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:48:56.153013 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:48:56.152979 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:49:04.454857 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:49:04.454824 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:49:13.956194 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:49:13.956155 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:49:24.959520 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:49:24.959484 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:49:34.255597 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:49:34.255502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:49:46.358184 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:49:46.358139 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:49:56.055430 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:49:56.055389 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:50:18.454413 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:50:18.454357 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:50:28.960048 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:50:28.960012 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:50:51.455740 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:50:51.455703 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:51:02.059903 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:51:02.059819 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:51:16.955245 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:51:16.955197 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:51:26.358045 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:51:26.358008 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:51:42.555645 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:51:42.555607 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:53:08.798222 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:53:08.798187 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:53:08.798222 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:53:08.798206 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:53:42.858350 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:53:42.858315 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:53:53.362132 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:53:53.362095 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:54:03.156030 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:54:03.155945 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:54:15.765242 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:54:15.765206 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:54:25.560356 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:54:25.560320 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:54:37.061221 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:54:37.061179 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:54:45.751316 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:54:45.751278 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:54:56.958251 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:54:56.958216 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:55:07.564134 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:55:07.564100 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:55:16.863526 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:55:16.863494 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:55:28.354645 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:55:28.354595 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:55:50.359693 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:55:50.359602 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:56:00.659002 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:56:00.658965 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:56:15.364436 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:56:15.364400 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:56:29.664545 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:56:29.664507 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:57:02.265967 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:57:02.265890 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:57:15.564118 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:57:15.564083 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:57:22.460433 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:57:22.460399 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:57:38.665562 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:57:38.665526 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:57:49.156448 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:57:49.156413 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:57:58.455769 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:57:58.455728 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:58:07.562146 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:58:07.562113 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:58:08.837148 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:58:08.837114 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:58:08.838932 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:58:08.838911 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 04:58:18.460114 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:58:18.460078 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:58:27.954243 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:58:27.954204 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:58:41.057193 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:58:41.057099 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:58:50.164025 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:58:50.163985 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:59:00.269839 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:59:00.269801 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:59:10.157288 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:59:10.157253 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:59:22.950925 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:59:22.950888 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:59:31.457339 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:59:31.457300 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:59:43.465602 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:59:43.465565 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 04:59:55.458824 ip-10-0-142-47 kubenswrapper[2572]: I0722 04:59:55.458790 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:00:00.131933 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.131895 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc"] Jul 22 05:00:00.132300 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.132266 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b844ba8d-2d5c-4680-a624-154807fe27bf" containerName="cleanup" Jul 22 05:00:00.132300 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.132277 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b844ba8d-2d5c-4680-a624-154807fe27bf" containerName="cleanup" Jul 22 05:00:00.132397 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.132346 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b844ba8d-2d5c-4680-a624-154807fe27bf" containerName="cleanup" Jul 22 05:00:00.135423 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.135406 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc" Jul 22 05:00:00.137796 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.137782 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:00:00.145207 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.145186 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc" Jul 22 05:00:00.147552 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.147528 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc"] Jul 22 05:00:00.274817 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:00.274792 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc"] Jul 22 05:00:00.277149 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:00:00.277120 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4254c65b_0c78_483d_af86_80415362df58.slice/crio-8381b4f8c7d86aadf320549812296010554fef4b53cd9ad61ce67bd16d23b79d WatchSource:0}: Error finding container 8381b4f8c7d86aadf320549812296010554fef4b53cd9ad61ce67bd16d23b79d: Status 404 returned error can't find the container with id 8381b4f8c7d86aadf320549812296010554fef4b53cd9ad61ce67bd16d23b79d Jul 22 05:00:01.209097 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:01.209065 2572 generic.go:356] "Generic (PLEG): container finished" podID="4254c65b-0c78-483d-af86-80415362df58" containerID="c2055ae332f14a5b74be88cdadbd2113dd227fc14a2e714c829d433f1590e298" exitCode=0 Jul 22 05:00:01.209446 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:01.209117 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc" event={"ID":"4254c65b-0c78-483d-af86-80415362df58","Type":"ContainerDied","Data":"c2055ae332f14a5b74be88cdadbd2113dd227fc14a2e714c829d433f1590e298"} Jul 22 05:00:01.209446 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:01.209159 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc" event={"ID":"4254c65b-0c78-483d-af86-80415362df58","Type":"ContainerStarted","Data":"8381b4f8c7d86aadf320549812296010554fef4b53cd9ad61ce67bd16d23b79d"} Jul 22 05:00:02.345848 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:02.345827 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc" Jul 22 05:00:03.220990 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:03.220900 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc" event={"ID":"4254c65b-0c78-483d-af86-80415362df58","Type":"ContainerDied","Data":"8381b4f8c7d86aadf320549812296010554fef4b53cd9ad61ce67bd16d23b79d"} Jul 22 05:00:03.220990 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:03.220950 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8381b4f8c7d86aadf320549812296010554fef4b53cd9ad61ce67bd16d23b79d" Jul 22 05:00:03.220990 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:03.220912 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744940-x8spc" Jul 22 05:00:04.664887 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:04.664855 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:00:18.661159 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:18.661122 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:00:27.786433 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:27.786392 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:00:37.967068 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:37.967034 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:00:49.059459 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:00:49.059408 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:01:06.654792 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.654761 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx"] Jul 22 05:01:06.655250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.655155 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4254c65b-0c78-483d-af86-80415362df58" containerName="cleanup" Jul 22 05:01:06.655250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.655166 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4254c65b-0c78-483d-af86-80415362df58" containerName="cleanup" Jul 22 05:01:06.655250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.655240 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4254c65b-0c78-483d-af86-80415362df58" containerName="cleanup" Jul 22 05:01:06.658362 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.658344 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" Jul 22 05:01:06.660667 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.660641 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:01:06.660806 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.660731 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:01:06.665529 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.665453 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx"] Jul 22 05:01:06.852447 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.852410 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69d6ac7-37a6-4045-84f6-1edfeb421717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx\" (UID: \"e69d6ac7-37a6-4045-84f6-1edfeb421717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" Jul 22 05:01:06.953859 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.953751 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69d6ac7-37a6-4045-84f6-1edfeb421717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx\" (UID: \"e69d6ac7-37a6-4045-84f6-1edfeb421717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" Jul 22 05:01:06.954527 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.954504 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69d6ac7-37a6-4045-84f6-1edfeb421717-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx\" (UID: \"e69d6ac7-37a6-4045-84f6-1edfeb421717\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" Jul 22 05:01:06.970730 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:06.970703 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" Jul 22 05:01:07.103314 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:07.103288 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx"] Jul 22 05:01:07.105091 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:01:07.105066 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode69d6ac7_37a6_4045_84f6_1edfeb421717.slice/crio-462a56b9e5f135bcb2481dd483624c4cbf9e9af049f5383f8667048a08c56d13 WatchSource:0}: Error finding container 462a56b9e5f135bcb2481dd483624c4cbf9e9af049f5383f8667048a08c56d13: Status 404 returned error can't find the container with id 462a56b9e5f135bcb2481dd483624c4cbf9e9af049f5383f8667048a08c56d13 Jul 22 05:01:07.560994 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:07.560955 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" event={"ID":"e69d6ac7-37a6-4045-84f6-1edfeb421717","Type":"ContainerStarted","Data":"65d385a8cc3949396b5e76e73ed921702b40249168d4900d0ceb57c0a756caaa"} Jul 22 05:01:07.561164 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:07.561013 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" event={"ID":"e69d6ac7-37a6-4045-84f6-1edfeb421717","Type":"ContainerStarted","Data":"462a56b9e5f135bcb2481dd483624c4cbf9e9af049f5383f8667048a08c56d13"} Jul 22 05:01:07.577012 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:07.576958 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" podStartSLOduration=1.576944225 podStartE2EDuration="1.576944225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:01:07.574559756 +0000 UTC m=+1679.293164894" watchObservedRunningTime="2026-07-22 05:01:07.576944225 +0000 UTC m=+1679.295549359" Jul 22 05:01:08.567716 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:08.567675 2572 generic.go:356] "Generic (PLEG): container finished" podID="e69d6ac7-37a6-4045-84f6-1edfeb421717" containerID="65d385a8cc3949396b5e76e73ed921702b40249168d4900d0ceb57c0a756caaa" exitCode=0 Jul 22 05:01:08.568095 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:08.567725 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" event={"ID":"e69d6ac7-37a6-4045-84f6-1edfeb421717","Type":"ContainerDied","Data":"65d385a8cc3949396b5e76e73ed921702b40249168d4900d0ceb57c0a756caaa"} Jul 22 05:01:09.705753 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:09.705728 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" Jul 22 05:01:09.778045 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:09.778013 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69d6ac7-37a6-4045-84f6-1edfeb421717-maas-api-service-ca\") pod \"e69d6ac7-37a6-4045-84f6-1edfeb421717\" (UID: \"e69d6ac7-37a6-4045-84f6-1edfeb421717\") " Jul 22 05:01:09.778360 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:09.778339 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e69d6ac7-37a6-4045-84f6-1edfeb421717-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e69d6ac7-37a6-4045-84f6-1edfeb421717" (UID: "e69d6ac7-37a6-4045-84f6-1edfeb421717"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:01:09.878892 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:09.878818 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e69d6ac7-37a6-4045-84f6-1edfeb421717-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:01:10.579623 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:10.579589 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" Jul 22 05:01:10.579792 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:10.579622 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx" event={"ID":"e69d6ac7-37a6-4045-84f6-1edfeb421717","Type":"ContainerDied","Data":"462a56b9e5f135bcb2481dd483624c4cbf9e9af049f5383f8667048a08c56d13"} Jul 22 05:01:10.579792 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:10.579669 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="462a56b9e5f135bcb2481dd483624c4cbf9e9af049f5383f8667048a08c56d13" Jul 22 05:01:19.157717 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:19.157686 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:01:21.734556 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:21.734514 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx"] Jul 22 05:01:21.738453 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:21.738424 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9e76c960-nnwwx"] Jul 22 05:01:22.800095 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:22.800061 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e69d6ac7-37a6-4045-84f6-1edfeb421717" path="/var/lib/kubelet/pods/e69d6ac7-37a6-4045-84f6-1edfeb421717/volumes" Jul 22 05:01:27.959553 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:27.959517 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:01:50.492485 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.492452 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2"] Jul 22 05:01:50.492909 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.492801 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e69d6ac7-37a6-4045-84f6-1edfeb421717" containerName="revoke-keys" Jul 22 05:01:50.492909 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.492813 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e69d6ac7-37a6-4045-84f6-1edfeb421717" containerName="revoke-keys" Jul 22 05:01:50.492909 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.492876 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e69d6ac7-37a6-4045-84f6-1edfeb421717" containerName="revoke-keys" Jul 22 05:01:50.502192 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.502153 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" Jul 22 05:01:50.503472 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.503443 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2"] Jul 22 05:01:50.505189 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.505061 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:01:50.505315 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.505253 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:01:50.624022 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.623988 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1108b45-82a7-4c7a-aead-7fc71ccdf273-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2\" (UID: \"b1108b45-82a7-4c7a-aead-7fc71ccdf273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" Jul 22 05:01:50.724473 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.724437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1108b45-82a7-4c7a-aead-7fc71ccdf273-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2\" (UID: \"b1108b45-82a7-4c7a-aead-7fc71ccdf273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" Jul 22 05:01:50.725020 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.725000 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1108b45-82a7-4c7a-aead-7fc71ccdf273-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2\" (UID: \"b1108b45-82a7-4c7a-aead-7fc71ccdf273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" Jul 22 05:01:50.812911 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.812882 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" Jul 22 05:01:50.949121 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:50.949092 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2"] Jul 22 05:01:50.950741 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:01:50.950716 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1108b45_82a7_4c7a_aead_7fc71ccdf273.slice/crio-18dbb50810482fca90ddc2d6adb45a64191189001fd4fb7a1395300dae78158a WatchSource:0}: Error finding container 18dbb50810482fca90ddc2d6adb45a64191189001fd4fb7a1395300dae78158a: Status 404 returned error can't find the container with id 18dbb50810482fca90ddc2d6adb45a64191189001fd4fb7a1395300dae78158a Jul 22 05:01:51.801408 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:51.801361 2572 generic.go:356] "Generic (PLEG): container finished" podID="b1108b45-82a7-4c7a-aead-7fc71ccdf273" containerID="430c7d5036f2d24dd2f1bda53bc21757aefca1ed4779c35b484a8d1a3aeefa04" exitCode=0 Jul 22 05:01:51.801775 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:51.801450 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" event={"ID":"b1108b45-82a7-4c7a-aead-7fc71ccdf273","Type":"ContainerDied","Data":"430c7d5036f2d24dd2f1bda53bc21757aefca1ed4779c35b484a8d1a3aeefa04"} Jul 22 05:01:51.801775 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:51.801503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" event={"ID":"b1108b45-82a7-4c7a-aead-7fc71ccdf273","Type":"ContainerStarted","Data":"18dbb50810482fca90ddc2d6adb45a64191189001fd4fb7a1395300dae78158a"} Jul 22 05:01:52.938692 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:52.938668 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" Jul 22 05:01:53.042206 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:53.042178 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1108b45-82a7-4c7a-aead-7fc71ccdf273-maas-api-service-ca\") pod \"b1108b45-82a7-4c7a-aead-7fc71ccdf273\" (UID: \"b1108b45-82a7-4c7a-aead-7fc71ccdf273\") " Jul 22 05:01:53.042598 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:53.042580 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1108b45-82a7-4c7a-aead-7fc71ccdf273-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1108b45-82a7-4c7a-aead-7fc71ccdf273" (UID: "b1108b45-82a7-4c7a-aead-7fc71ccdf273"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:01:53.143851 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:53.143764 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1108b45-82a7-4c7a-aead-7fc71ccdf273-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:01:53.813418 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:53.813368 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" Jul 22 05:01:53.813418 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:53.813414 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2" event={"ID":"b1108b45-82a7-4c7a-aead-7fc71ccdf273","Type":"ContainerDied","Data":"18dbb50810482fca90ddc2d6adb45a64191189001fd4fb7a1395300dae78158a"} Jul 22 05:01:53.813613 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:01:53.813458 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18dbb50810482fca90ddc2d6adb45a64191189001fd4fb7a1395300dae78158a" Jul 22 05:02:03.460235 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:03.460192 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:02:05.573307 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:05.573270 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2"] Jul 22 05:02:05.575349 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:05.575328 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dec153cb-zxqh2"] Jul 22 05:02:06.800648 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:06.800619 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1108b45-82a7-4c7a-aead-7fc71ccdf273" path="/var/lib/kubelet/pods/b1108b45-82a7-4c7a-aead-7fc71ccdf273/volumes" Jul 22 05:02:16.354950 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:16.354913 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:02:34.348914 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.348874 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j"] Jul 22 05:02:34.349309 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.349225 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1108b45-82a7-4c7a-aead-7fc71ccdf273" containerName="revoke-keys" Jul 22 05:02:34.349309 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.349236 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1108b45-82a7-4c7a-aead-7fc71ccdf273" containerName="revoke-keys" Jul 22 05:02:34.349309 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.349304 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1108b45-82a7-4c7a-aead-7fc71ccdf273" containerName="revoke-keys" Jul 22 05:02:34.352250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.352233 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" Jul 22 05:02:34.354454 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.354433 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:02:34.354568 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.354463 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:02:34.361775 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.361740 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j"] Jul 22 05:02:34.386608 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.386584 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b257e79d-a014-4405-893d-4f93b9bbe4d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-303fe40d-gk84j\" (UID: \"b257e79d-a014-4405-893d-4f93b9bbe4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" Jul 22 05:02:34.487338 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.487307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b257e79d-a014-4405-893d-4f93b9bbe4d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-303fe40d-gk84j\" (UID: \"b257e79d-a014-4405-893d-4f93b9bbe4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" Jul 22 05:02:34.487928 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.487907 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b257e79d-a014-4405-893d-4f93b9bbe4d5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-303fe40d-gk84j\" (UID: \"b257e79d-a014-4405-893d-4f93b9bbe4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" Jul 22 05:02:34.671535 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.671445 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" Jul 22 05:02:34.802196 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:34.802167 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j"] Jul 22 05:02:34.803803 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:02:34.803775 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb257e79d_a014_4405_893d_4f93b9bbe4d5.slice/crio-63ea5f2a152a6da5bafa9410bb2f75469dbb0b82abe26933723ac65f2f27625f WatchSource:0}: Error finding container 63ea5f2a152a6da5bafa9410bb2f75469dbb0b82abe26933723ac65f2f27625f: Status 404 returned error can't find the container with id 63ea5f2a152a6da5bafa9410bb2f75469dbb0b82abe26933723ac65f2f27625f Jul 22 05:02:35.030863 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:35.030834 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" event={"ID":"b257e79d-a014-4405-893d-4f93b9bbe4d5","Type":"ContainerStarted","Data":"63ea5f2a152a6da5bafa9410bb2f75469dbb0b82abe26933723ac65f2f27625f"} Jul 22 05:02:36.037427 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:36.037395 2572 generic.go:356] "Generic (PLEG): container finished" podID="b257e79d-a014-4405-893d-4f93b9bbe4d5" containerID="df05e2711e6150c0a8f8801710917b871a06d91190f24d2c6f686205465f48cd" exitCode=0 Jul 22 05:02:36.037789 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:36.037481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" event={"ID":"b257e79d-a014-4405-893d-4f93b9bbe4d5","Type":"ContainerDied","Data":"df05e2711e6150c0a8f8801710917b871a06d91190f24d2c6f686205465f48cd"} Jul 22 05:02:37.171047 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:37.171022 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" Jul 22 05:02:37.210666 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:37.210642 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b257e79d-a014-4405-893d-4f93b9bbe4d5-maas-api-service-ca\") pod \"b257e79d-a014-4405-893d-4f93b9bbe4d5\" (UID: \"b257e79d-a014-4405-893d-4f93b9bbe4d5\") " Jul 22 05:02:37.210961 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:37.210940 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b257e79d-a014-4405-893d-4f93b9bbe4d5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b257e79d-a014-4405-893d-4f93b9bbe4d5" (UID: "b257e79d-a014-4405-893d-4f93b9bbe4d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:02:37.311507 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:37.311441 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b257e79d-a014-4405-893d-4f93b9bbe4d5-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:02:38.048601 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:38.048573 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" Jul 22 05:02:38.048773 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:38.048607 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j" event={"ID":"b257e79d-a014-4405-893d-4f93b9bbe4d5","Type":"ContainerDied","Data":"63ea5f2a152a6da5bafa9410bb2f75469dbb0b82abe26933723ac65f2f27625f"} Jul 22 05:02:38.048773 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:38.048655 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63ea5f2a152a6da5bafa9410bb2f75469dbb0b82abe26933723ac65f2f27625f" Jul 22 05:02:47.559418 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:47.559369 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:02:49.429370 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:49.429339 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j"] Jul 22 05:02:49.433388 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:49.433347 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-303fe40d-gk84j"] Jul 22 05:02:50.801869 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:02:50.801828 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b257e79d-a014-4405-893d-4f93b9bbe4d5" path="/var/lib/kubelet/pods/b257e79d-a014-4405-893d-4f93b9bbe4d5/volumes" Jul 22 05:03:00.360027 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:00.359992 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:03:08.881952 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:08.881910 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:03:08.883319 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:08.883300 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:03:18.702217 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.702179 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl"] Jul 22 05:03:18.702775 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.702755 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b257e79d-a014-4405-893d-4f93b9bbe4d5" containerName="revoke-keys" Jul 22 05:03:18.702821 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.702780 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b257e79d-a014-4405-893d-4f93b9bbe4d5" containerName="revoke-keys" Jul 22 05:03:18.702863 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.702853 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b257e79d-a014-4405-893d-4f93b9bbe4d5" containerName="revoke-keys" Jul 22 05:03:18.705905 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.705887 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" Jul 22 05:03:18.708631 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.708597 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:03:18.709036 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.709016 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:03:18.710870 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.710839 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl"] Jul 22 05:03:18.762611 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.762582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7db36d-d3f0-4be3-8431-b648fae7ce6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl\" (UID: \"6d7db36d-d3f0-4be3-8431-b648fae7ce6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" Jul 22 05:03:18.863526 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.863488 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7db36d-d3f0-4be3-8431-b648fae7ce6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl\" (UID: \"6d7db36d-d3f0-4be3-8431-b648fae7ce6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" Jul 22 05:03:18.864189 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:18.864156 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7db36d-d3f0-4be3-8431-b648fae7ce6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl\" (UID: \"6d7db36d-d3f0-4be3-8431-b648fae7ce6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" Jul 22 05:03:19.017509 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:19.017478 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" Jul 22 05:03:19.358257 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:03:19.358224 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7db36d_d3f0_4be3_8431_b648fae7ce6f.slice/crio-f85275a1268fbc4684bd0072414b920e40300a3b2b8b02908ecdc7fe7711af09 WatchSource:0}: Error finding container f85275a1268fbc4684bd0072414b920e40300a3b2b8b02908ecdc7fe7711af09: Status 404 returned error can't find the container with id f85275a1268fbc4684bd0072414b920e40300a3b2b8b02908ecdc7fe7711af09 Jul 22 05:03:19.358465 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:19.358444 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl"] Jul 22 05:03:20.275492 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:20.275457 2572 generic.go:356] "Generic (PLEG): container finished" podID="6d7db36d-d3f0-4be3-8431-b648fae7ce6f" containerID="94e4f9788a7010ffeaee59b68aa434ec9f55a58aeb24e3a0311f4a89e1075f6b" exitCode=0 Jul 22 05:03:20.275855 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:20.275526 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" event={"ID":"6d7db36d-d3f0-4be3-8431-b648fae7ce6f","Type":"ContainerDied","Data":"94e4f9788a7010ffeaee59b68aa434ec9f55a58aeb24e3a0311f4a89e1075f6b"} Jul 22 05:03:20.275855 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:20.275568 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" event={"ID":"6d7db36d-d3f0-4be3-8431-b648fae7ce6f","Type":"ContainerStarted","Data":"f85275a1268fbc4684bd0072414b920e40300a3b2b8b02908ecdc7fe7711af09"} Jul 22 05:03:21.413370 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:21.413343 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" Jul 22 05:03:21.582527 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:21.582457 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7db36d-d3f0-4be3-8431-b648fae7ce6f-maas-api-service-ca\") pod \"6d7db36d-d3f0-4be3-8431-b648fae7ce6f\" (UID: \"6d7db36d-d3f0-4be3-8431-b648fae7ce6f\") " Jul 22 05:03:21.582769 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:21.582748 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7db36d-d3f0-4be3-8431-b648fae7ce6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d7db36d-d3f0-4be3-8431-b648fae7ce6f" (UID: "6d7db36d-d3f0-4be3-8431-b648fae7ce6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:03:21.682965 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:21.682940 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7db36d-d3f0-4be3-8431-b648fae7ce6f-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:03:22.287325 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:22.287284 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" event={"ID":"6d7db36d-d3f0-4be3-8431-b648fae7ce6f","Type":"ContainerDied","Data":"f85275a1268fbc4684bd0072414b920e40300a3b2b8b02908ecdc7fe7711af09"} Jul 22 05:03:22.287523 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:22.287293 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl" Jul 22 05:03:22.287523 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:22.287371 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f85275a1268fbc4684bd0072414b920e40300a3b2b8b02908ecdc7fe7711af09" Jul 22 05:03:32.058329 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:32.058293 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:03:33.776835 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:33.776801 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl"] Jul 22 05:03:33.778818 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:33.778796 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e622b09-fn4cl"] Jul 22 05:03:34.801215 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:03:34.801182 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d7db36d-d3f0-4be3-8431-b648fae7ce6f" path="/var/lib/kubelet/pods/6d7db36d-d3f0-4be3-8431-b648fae7ce6f/volumes" Jul 22 05:04:58.564654 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:04:58.564616 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:05:08.154575 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:08.154537 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:05:15.855895 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:15.855859 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:05:24.816004 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.815972 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2"] Jul 22 05:05:24.816395 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.816326 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d7db36d-d3f0-4be3-8431-b648fae7ce6f" containerName="revoke-keys" Jul 22 05:05:24.816395 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.816337 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d7db36d-d3f0-4be3-8431-b648fae7ce6f" containerName="revoke-keys" Jul 22 05:05:24.816480 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.816413 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d7db36d-d3f0-4be3-8431-b648fae7ce6f" containerName="revoke-keys" Jul 22 05:05:24.819558 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.819541 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" Jul 22 05:05:24.821937 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.821913 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:05:24.822072 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.821973 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:05:24.826653 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.826628 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2"] Jul 22 05:05:24.938364 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:24.938333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72973b9e-28f6-4db1-9126-4cffb0640dc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2\" (UID: \"72973b9e-28f6-4db1-9126-4cffb0640dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" Jul 22 05:05:25.039333 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:25.039300 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72973b9e-28f6-4db1-9126-4cffb0640dc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2\" (UID: \"72973b9e-28f6-4db1-9126-4cffb0640dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" Jul 22 05:05:25.039895 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:25.039877 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72973b9e-28f6-4db1-9126-4cffb0640dc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2\" (UID: \"72973b9e-28f6-4db1-9126-4cffb0640dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" Jul 22 05:05:25.131532 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:25.131454 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" Jul 22 05:05:25.260708 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:25.260685 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2"] Jul 22 05:05:25.262555 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:05:25.262528 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72973b9e_28f6_4db1_9126_4cffb0640dc0.slice/crio-287896aa60094097c5122c8a0c73f232a35993c4c7be6a99289c609b482a83cf WatchSource:0}: Error finding container 287896aa60094097c5122c8a0c73f232a35993c4c7be6a99289c609b482a83cf: Status 404 returned error can't find the container with id 287896aa60094097c5122c8a0c73f232a35993c4c7be6a99289c609b482a83cf Jul 22 05:05:25.969759 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:25.969729 2572 generic.go:356] "Generic (PLEG): container finished" podID="72973b9e-28f6-4db1-9126-4cffb0640dc0" containerID="8e1523e4a556e4acd4af88921014707391182afc8108182fe4b126cae8d2b5e7" exitCode=0 Jul 22 05:05:25.970097 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:25.969781 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" event={"ID":"72973b9e-28f6-4db1-9126-4cffb0640dc0","Type":"ContainerDied","Data":"8e1523e4a556e4acd4af88921014707391182afc8108182fe4b126cae8d2b5e7"} Jul 22 05:05:25.970097 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:25.969868 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" event={"ID":"72973b9e-28f6-4db1-9126-4cffb0640dc0","Type":"ContainerStarted","Data":"287896aa60094097c5122c8a0c73f232a35993c4c7be6a99289c609b482a83cf"} Jul 22 05:05:27.105673 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:27.105648 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" Jul 22 05:05:27.156632 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:27.156603 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72973b9e-28f6-4db1-9126-4cffb0640dc0-maas-api-service-ca\") pod \"72973b9e-28f6-4db1-9126-4cffb0640dc0\" (UID: \"72973b9e-28f6-4db1-9126-4cffb0640dc0\") " Jul 22 05:05:27.156897 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:27.156877 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72973b9e-28f6-4db1-9126-4cffb0640dc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72973b9e-28f6-4db1-9126-4cffb0640dc0" (UID: "72973b9e-28f6-4db1-9126-4cffb0640dc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:05:27.258086 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:27.258019 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72973b9e-28f6-4db1-9126-4cffb0640dc0-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:05:27.982547 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:27.982510 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" event={"ID":"72973b9e-28f6-4db1-9126-4cffb0640dc0","Type":"ContainerDied","Data":"287896aa60094097c5122c8a0c73f232a35993c4c7be6a99289c609b482a83cf"} Jul 22 05:05:27.982721 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:27.982558 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="287896aa60094097c5122c8a0c73f232a35993c4c7be6a99289c609b482a83cf" Jul 22 05:05:27.982721 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:27.982522 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2" Jul 22 05:05:37.857131 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:37.857098 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:05:39.888746 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:39.888712 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2"] Jul 22 05:05:39.891986 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:39.891960 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-31b554fe-l6mh2"] Jul 22 05:05:40.800420 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:05:40.800368 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72973b9e-28f6-4db1-9126-4cffb0640dc0" path="/var/lib/kubelet/pods/72973b9e-28f6-4db1-9126-4cffb0640dc0/volumes" Jul 22 05:06:15.555968 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:15.555880 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:06:21.563744 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.563707 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4"] Jul 22 05:06:21.564123 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.564110 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72973b9e-28f6-4db1-9126-4cffb0640dc0" containerName="revoke-keys" Jul 22 05:06:21.564167 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.564127 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="72973b9e-28f6-4db1-9126-4cffb0640dc0" containerName="revoke-keys" Jul 22 05:06:21.564223 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.564212 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="72973b9e-28f6-4db1-9126-4cffb0640dc0" containerName="revoke-keys" Jul 22 05:06:21.567428 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.567412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.569666 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.569643 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-06f9a996\"/\"kube-root-ca.crt\"" Jul 22 05:06:21.570629 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.570597 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-06f9a996\"/\"default-dockercfg-ssxf9\"" Jul 22 05:06:21.570629 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.570618 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-06f9a996\"/\"e2e-lifecycle-model-06f9a996-kserve-self-signed-certs\"" Jul 22 05:06:21.570629 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.570618 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-06f9a996\"/\"openshift-service-ca.crt\"" Jul 22 05:06:21.579628 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.579606 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4"] Jul 22 05:06:21.601213 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.601187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-home\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.601318 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.601234 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-kube-api-access-qwdp2\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.601318 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.601280 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-dshm\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.601416 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.601319 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-model-cache\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.601416 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.601358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tmp-dir\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.601416 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.601405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tls-certs\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.702810 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.702777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-model-cache\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.702974 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.702822 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tmp-dir\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.702974 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.702845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tls-certs\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.702974 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.702886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-home\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.702974 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.702917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-kube-api-access-qwdp2\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.702974 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.702942 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-dshm\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.703254 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.703229 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tmp-dir\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.703316 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.703284 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-model-cache\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.703362 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.703307 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-home\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.705183 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.705164 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-dshm\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.705350 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.705331 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tls-certs\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.711074 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.711057 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-kube-api-access-qwdp2\") pod \"e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:21.876769 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:21.876688 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:22.005039 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:22.005011 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4"] Jul 22 05:06:22.006496 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:06:22.006470 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31d2d8d6_9ff9_4d7d_a816_b1235a6f7c53.slice/crio-91e4293aa88b2f9f12754cc8a8bf6a2b9bf9d71a21bc71b8952d8843ab2c6183 WatchSource:0}: Error finding container 91e4293aa88b2f9f12754cc8a8bf6a2b9bf9d71a21bc71b8952d8843ab2c6183: Status 404 returned error can't find the container with id 91e4293aa88b2f9f12754cc8a8bf6a2b9bf9d71a21bc71b8952d8843ab2c6183 Jul 22 05:06:22.288661 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:22.288624 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" event={"ID":"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53","Type":"ContainerStarted","Data":"737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714"} Jul 22 05:06:22.288661 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:22.288664 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" event={"ID":"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53","Type":"ContainerStarted","Data":"91e4293aa88b2f9f12754cc8a8bf6a2b9bf9d71a21bc71b8952d8843ab2c6183"} Jul 22 05:06:22.306625 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:22.306579 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" podStartSLOduration=1.306565012 podStartE2EDuration="1.306565012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:06:22.304132148 +0000 UTC m=+1994.022737283" watchObservedRunningTime="2026-07-22 05:06:22.306565012 +0000 UTC m=+1994.025170147" Jul 22 05:06:26.464838 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:26.464804 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:06:31.877466 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:31.877428 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:31.877830 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:31.877481 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:31.889734 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:31.889711 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:32.353747 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:32.353720 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:06:58.057182 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:06:58.057142 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:07:08.934776 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:08.934746 2572 scope.go:117] "RemoveContainer" containerID="65d385a8cc3949396b5e76e73ed921702b40249168d4900d0ceb57c0a756caaa" Jul 22 05:07:15.851606 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:15.851572 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:07:26.759033 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:26.758997 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:07:28.971820 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:28.971786 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4"] Jul 22 05:07:43.994092 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:43.994031 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" podUID="31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" containerName="main" containerID="cri-o://737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714" gracePeriod=45 Jul 22 05:07:44.237535 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.237509 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:07:44.380219 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380185 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-kube-api-access-qwdp2\") pod \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " Jul 22 05:07:44.380402 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380240 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-model-cache\") pod \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " Jul 22 05:07:44.380402 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380283 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-home\") pod \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " Jul 22 05:07:44.380402 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380299 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tmp-dir\") pod \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " Jul 22 05:07:44.380402 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380353 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tls-certs\") pod \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " Jul 22 05:07:44.380649 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380454 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-dshm\") pod \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\" (UID: \"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53\") " Jul 22 05:07:44.380649 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380508 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-model-cache" (OuterVolumeSpecName: "model-cache") pod "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" (UID: "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:07:44.380649 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380546 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-home" (OuterVolumeSpecName: "home") pod "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" (UID: "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:07:44.380649 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380586 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" (UID: "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:07:44.380845 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380772 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-model-cache\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:07:44.380845 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380791 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-home\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:07:44.380845 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.380805 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tmp-dir\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:07:44.382419 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.382370 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-kube-api-access-qwdp2" (OuterVolumeSpecName: "kube-api-access-qwdp2") pod "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" (UID: "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53"). InnerVolumeSpecName "kube-api-access-qwdp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:07:44.382554 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.382503 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-dshm" (OuterVolumeSpecName: "dshm") pod "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" (UID: "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:07:44.382741 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.382723 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" (UID: "31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 05:07:44.481496 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.481456 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-dshm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:07:44.481496 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.481492 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdp2\" (UniqueName: \"kubernetes.io/projected/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-kube-api-access-qwdp2\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:07:44.481496 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.481503 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53-tls-certs\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:07:44.734318 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.734230 2572 generic.go:356] "Generic (PLEG): container finished" podID="31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" containerID="737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714" exitCode=0 Jul 22 05:07:44.734318 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.734275 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" Jul 22 05:07:44.734318 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.734299 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" event={"ID":"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53","Type":"ContainerDied","Data":"737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714"} Jul 22 05:07:44.734646 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.734339 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4" event={"ID":"31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53","Type":"ContainerDied","Data":"91e4293aa88b2f9f12754cc8a8bf6a2b9bf9d71a21bc71b8952d8843ab2c6183"} Jul 22 05:07:44.734646 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.734360 2572 scope.go:117] "RemoveContainer" containerID="737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714" Jul 22 05:07:44.744608 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.744587 2572 scope.go:117] "RemoveContainer" containerID="737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714" Jul 22 05:07:44.744873 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:07:44.744851 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714\": container with ID starting with 737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714 not found: ID does not exist" containerID="737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714" Jul 22 05:07:44.744962 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.744878 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714"} err="failed to get container status \"737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714\": rpc error: code = NotFound desc = could not find container \"737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714\": container with ID starting with 737cac452ef51919bde4c9e9b9251fe5365f57f4f086c605a9f12e809c2ee714 not found: ID does not exist" Jul 22 05:07:44.755039 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.755018 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4"] Jul 22 05:07:44.758981 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.758959 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-06f9a996/e2e-lifecycle-model-06f9a996-kserve-6f49765f6c-h5jb4"] Jul 22 05:07:44.800333 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:07:44.800311 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" path="/var/lib/kubelet/pods/31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53/volumes" Jul 22 05:08:08.917833 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:08:08.917805 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:08:08.920163 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:08:08.920139 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:08:08.957181 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:08:08.957151 2572 scope.go:117] "RemoveContainer" containerID="430c7d5036f2d24dd2f1bda53bc21757aefca1ed4779c35b484a8d1a3aeefa04" Jul 22 05:08:16.058271 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:08:16.058233 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:08:27.855350 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:08:27.855315 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:09:08.982108 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:08.982025 2572 scope.go:117] "RemoveContainer" containerID="df05e2711e6150c0a8f8801710917b871a06d91190f24d2c6f686205465f48cd" Jul 22 05:09:15.854198 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:15.854163 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:09:32.061868 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:32.061831 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:09:36.082781 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.082748 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd"] Jul 22 05:09:36.083171 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.083156 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" containerName="main" Jul 22 05:09:36.083248 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.083174 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" containerName="main" Jul 22 05:09:36.083248 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.083246 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d2d8d6-9ff9-4d7d-a816-b1235a6f7c53" containerName="main" Jul 22 05:09:36.086535 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.086517 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.089686 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.089665 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fce892\"/\"openshift-service-ca.crt\"" Jul 22 05:09:36.089686 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.089679 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fce892\"/\"default-dockercfg-4f9s6\"" Jul 22 05:09:36.089854 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.089689 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fce892\"/\"test-model-fce892-kserve-self-signed-certs\"" Jul 22 05:09:36.089854 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.089679 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fce892\"/\"kube-root-ca.crt\"" Jul 22 05:09:36.094246 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.094165 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd"] Jul 22 05:09:36.201730 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.201700 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-model-cache\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.201898 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.201735 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16971041-1519-4c28-9dd5-02d5f09fa2be-tls-certs\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.201898 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.201862 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxsm\" (UniqueName: \"kubernetes.io/projected/16971041-1519-4c28-9dd5-02d5f09fa2be-kube-api-access-fhxsm\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.202036 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.201910 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-home\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.202036 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.201935 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-dshm\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.202036 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.201976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-tmp-dir\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.302750 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.302717 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxsm\" (UniqueName: \"kubernetes.io/projected/16971041-1519-4c28-9dd5-02d5f09fa2be-kube-api-access-fhxsm\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.302897 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.302758 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-home\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.302897 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.302777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-dshm\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.302897 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.302802 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-tmp-dir\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.302897 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.302829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-model-cache\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.302897 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.302844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16971041-1519-4c28-9dd5-02d5f09fa2be-tls-certs\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.303203 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.303185 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-model-cache\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.303285 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.303210 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-tmp-dir\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.303285 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.303182 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-home\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.305207 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.305177 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-dshm\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.305386 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.305356 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16971041-1519-4c28-9dd5-02d5f09fa2be-tls-certs\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.313260 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.313241 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxsm\" (UniqueName: \"kubernetes.io/projected/16971041-1519-4c28-9dd5-02d5f09fa2be-kube-api-access-fhxsm\") pod \"test-model-fce892-kserve-7484887587-gm8nd\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.397570 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.397493 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:36.526876 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:36.526852 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd"] Jul 22 05:09:36.528809 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:09:36.528780 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16971041_1519_4c28_9dd5_02d5f09fa2be.slice/crio-ccd7a3e30f66b28febc3e48b256ae5c707d9f2d47022cd589a5e46f4638723b0 WatchSource:0}: Error finding container ccd7a3e30f66b28febc3e48b256ae5c707d9f2d47022cd589a5e46f4638723b0: Status 404 returned error can't find the container with id ccd7a3e30f66b28febc3e48b256ae5c707d9f2d47022cd589a5e46f4638723b0 Jul 22 05:09:37.333340 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:37.333295 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" event={"ID":"16971041-1519-4c28-9dd5-02d5f09fa2be","Type":"ContainerStarted","Data":"de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a"} Jul 22 05:09:37.333340 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:37.333342 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" event={"ID":"16971041-1519-4c28-9dd5-02d5f09fa2be","Type":"ContainerStarted","Data":"ccd7a3e30f66b28febc3e48b256ae5c707d9f2d47022cd589a5e46f4638723b0"} Jul 22 05:09:37.353664 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:37.353619 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" podStartSLOduration=1.3536037890000001 podStartE2EDuration="1.353603789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:09:37.351416454 +0000 UTC m=+2189.070021594" watchObservedRunningTime="2026-07-22 05:09:37.353603789 +0000 UTC m=+2189.072208944" Jul 22 05:09:41.864695 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:41.864656 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:09:46.397861 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:46.397826 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:46.398237 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:46.397883 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:46.410438 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:46.410411 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:47.400661 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:47.400630 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:09:57.960580 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:57.960548 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4"] Jul 22 05:09:57.964047 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:57.964023 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:57.966418 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:57.966398 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8e4aa9\"/\"openshift-service-ca.crt\"" Jul 22 05:09:57.967360 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:57.967334 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8e4aa9\"/\"default-dockercfg-k5cgc\"" Jul 22 05:09:57.967515 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:57.967393 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8e4aa9\"/\"kube-root-ca.crt\"" Jul 22 05:09:57.967515 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:57.967438 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8e4aa9\"/\"test-model-8e4aa9-kserve-self-signed-certs\"" Jul 22 05:09:57.972350 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:57.972329 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4"] Jul 22 05:09:58.094891 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.094841 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-model-cache\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.094891 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.094892 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-home\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.095106 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.094915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-tmp-dir\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.095106 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.094981 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjjh\" (UniqueName: \"kubernetes.io/projected/bc062c83-8be6-4677-831b-3a28bce6cd9d-kube-api-access-scjjh\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.095106 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.095064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-dshm\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.095106 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.095086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc062c83-8be6-4677-831b-3a28bce6cd9d-tls-certs\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196091 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjjh\" (UniqueName: \"kubernetes.io/projected/bc062c83-8be6-4677-831b-3a28bce6cd9d-kube-api-access-scjjh\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196258 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196126 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-dshm\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196258 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc062c83-8be6-4677-831b-3a28bce6cd9d-tls-certs\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196258 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-model-cache\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196258 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-home\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196258 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-tmp-dir\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196653 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196629 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-model-cache\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196724 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196675 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-tmp-dir\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.196724 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.196705 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-home\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.198480 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.198455 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-dshm\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.198749 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.198732 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc062c83-8be6-4677-831b-3a28bce6cd9d-tls-certs\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.203250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.203230 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjjh\" (UniqueName: \"kubernetes.io/projected/bc062c83-8be6-4677-831b-3a28bce6cd9d-kube-api-access-scjjh\") pod \"test-model-8e4aa9-kserve-58b4c487f7-2wdc4\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.275313 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.275285 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:09:58.405986 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.405959 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4"] Jul 22 05:09:58.407333 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:09:58.407306 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc062c83_8be6_4677_831b_3a28bce6cd9d.slice/crio-77d0e7490ee423916b85effcb7307277edd662d03aebc9da665fc5ed8a2d65d0 WatchSource:0}: Error finding container 77d0e7490ee423916b85effcb7307277edd662d03aebc9da665fc5ed8a2d65d0: Status 404 returned error can't find the container with id 77d0e7490ee423916b85effcb7307277edd662d03aebc9da665fc5ed8a2d65d0 Jul 22 05:09:58.445737 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:58.445709 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" event={"ID":"bc062c83-8be6-4677-831b-3a28bce6cd9d","Type":"ContainerStarted","Data":"77d0e7490ee423916b85effcb7307277edd662d03aebc9da665fc5ed8a2d65d0"} Jul 22 05:09:59.451673 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:59.451628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" event={"ID":"bc062c83-8be6-4677-831b-3a28bce6cd9d","Type":"ContainerStarted","Data":"4df4102843fd35d189f99fc4d0bb9011aa1903474d8a8d83160c9fec81fd906b"} Jul 22 05:09:59.466964 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:09:59.466905 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" podStartSLOduration=2.466887433 podStartE2EDuration="2.466887433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:09:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:09:59.466201046 +0000 UTC m=+2211.184806219" watchObservedRunningTime="2026-07-22 05:09:59.466887433 +0000 UTC m=+2211.185492569" Jul 22 05:10:03.557128 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:03.557089 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:10:08.276273 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:08.276234 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:10:08.276788 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:08.276291 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:10:08.288893 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:08.288863 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:10:08.509629 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:08.509602 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:10:09.005451 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:09.005418 2572 scope.go:117] "RemoveContainer" containerID="94e4f9788a7010ffeaee59b68aa434ec9f55a58aeb24e3a0311f4a89e1075f6b" Jul 22 05:10:42.667272 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:42.667238 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd"] Jul 22 05:10:43.561183 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.561151 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4"] Jul 22 05:10:43.707621 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.707588 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f"] Jul 22 05:10:43.711062 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.711042 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" Jul 22 05:10:43.713305 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.713278 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:10:43.713452 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.713314 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:10:43.718906 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.718502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f"] Jul 22 05:10:43.887673 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.887589 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fce892-qs84f\" (UID: \"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" Jul 22 05:10:43.988306 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.988270 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fce892-qs84f\" (UID: \"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" Jul 22 05:10:43.988903 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:43.988882 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fce892-qs84f\" (UID: \"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" Jul 22 05:10:44.022718 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:44.022694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" Jul 22 05:10:44.150745 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:44.150720 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f"] Jul 22 05:10:44.152346 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:10:44.152320 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bc9c65_8bcb_4ad0_9f53_bfd0ec68b918.slice/crio-5f5b167f48164bbf91e441e55e19e68ff71d2e5cf07fa1b629afa9154489c991 WatchSource:0}: Error finding container 5f5b167f48164bbf91e441e55e19e68ff71d2e5cf07fa1b629afa9154489c991: Status 404 returned error can't find the container with id 5f5b167f48164bbf91e441e55e19e68ff71d2e5cf07fa1b629afa9154489c991 Jul 22 05:10:44.698464 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:44.698332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" event={"ID":"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918","Type":"ContainerStarted","Data":"f8409242a2e83343344ee618670a7e31daed900166bd53cab735a67502238029"} Jul 22 05:10:44.698464 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:44.698420 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" event={"ID":"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918","Type":"ContainerStarted","Data":"5f5b167f48164bbf91e441e55e19e68ff71d2e5cf07fa1b629afa9154489c991"} Jul 22 05:10:44.714771 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:44.714723 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" podStartSLOduration=1.714707898 podStartE2EDuration="1.714707898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:10:44.712428626 +0000 UTC m=+2256.431033763" watchObservedRunningTime="2026-07-22 05:10:44.714707898 +0000 UTC m=+2256.433313032" Jul 22 05:10:45.704837 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:45.704798 2572 generic.go:356] "Generic (PLEG): container finished" podID="d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918" containerID="f8409242a2e83343344ee618670a7e31daed900166bd53cab735a67502238029" exitCode=0 Jul 22 05:10:45.705012 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:45.704847 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" event={"ID":"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918","Type":"ContainerDied","Data":"f8409242a2e83343344ee618670a7e31daed900166bd53cab735a67502238029"} Jul 22 05:10:46.846412 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:46.846363 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" Jul 22 05:10:47.014911 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:47.014883 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918-maas-api-service-ca\") pod \"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918\" (UID: \"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918\") " Jul 22 05:10:47.015288 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:47.015268 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918" (UID: "d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:10:47.116596 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:47.116566 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:47.260288 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:47.260251 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:10:47.718796 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:47.718767 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" Jul 22 05:10:47.718961 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:47.718804 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f" event={"ID":"d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918","Type":"ContainerDied","Data":"5f5b167f48164bbf91e441e55e19e68ff71d2e5cf07fa1b629afa9154489c991"} Jul 22 05:10:47.718961 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:47.718852 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f5b167f48164bbf91e441e55e19e68ff71d2e5cf07fa1b629afa9154489c991" Jul 22 05:10:57.693317 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:57.693248 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" podUID="16971041-1519-4c28-9dd5-02d5f09fa2be" containerName="main" containerID="cri-o://de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a" gracePeriod=45 Jul 22 05:10:57.952410 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:57.952337 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:10:58.000150 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000119 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16971041-1519-4c28-9dd5-02d5f09fa2be-tls-certs\") pod \"16971041-1519-4c28-9dd5-02d5f09fa2be\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " Jul 22 05:10:58.000307 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000155 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-home\") pod \"16971041-1519-4c28-9dd5-02d5f09fa2be\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " Jul 22 05:10:58.000307 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000177 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-tmp-dir\") pod \"16971041-1519-4c28-9dd5-02d5f09fa2be\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " Jul 22 05:10:58.000307 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000216 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhxsm\" (UniqueName: \"kubernetes.io/projected/16971041-1519-4c28-9dd5-02d5f09fa2be-kube-api-access-fhxsm\") pod \"16971041-1519-4c28-9dd5-02d5f09fa2be\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " Jul 22 05:10:58.000307 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000292 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-dshm\") pod \"16971041-1519-4c28-9dd5-02d5f09fa2be\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " Jul 22 05:10:58.000565 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000344 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-model-cache\") pod \"16971041-1519-4c28-9dd5-02d5f09fa2be\" (UID: \"16971041-1519-4c28-9dd5-02d5f09fa2be\") " Jul 22 05:10:58.000565 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000536 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-home" (OuterVolumeSpecName: "home") pod "16971041-1519-4c28-9dd5-02d5f09fa2be" (UID: "16971041-1519-4c28-9dd5-02d5f09fa2be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:10:58.000565 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000550 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16971041-1519-4c28-9dd5-02d5f09fa2be" (UID: "16971041-1519-4c28-9dd5-02d5f09fa2be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:10:58.000765 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000739 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-home\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:58.000836 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000769 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-tmp-dir\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:58.000836 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.000736 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-model-cache" (OuterVolumeSpecName: "model-cache") pod "16971041-1519-4c28-9dd5-02d5f09fa2be" (UID: "16971041-1519-4c28-9dd5-02d5f09fa2be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:10:58.002487 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.002464 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-dshm" (OuterVolumeSpecName: "dshm") pod "16971041-1519-4c28-9dd5-02d5f09fa2be" (UID: "16971041-1519-4c28-9dd5-02d5f09fa2be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:10:58.002588 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.002560 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16971041-1519-4c28-9dd5-02d5f09fa2be-kube-api-access-fhxsm" (OuterVolumeSpecName: "kube-api-access-fhxsm") pod "16971041-1519-4c28-9dd5-02d5f09fa2be" (UID: "16971041-1519-4c28-9dd5-02d5f09fa2be"). InnerVolumeSpecName "kube-api-access-fhxsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:10:58.002645 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.002611 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16971041-1519-4c28-9dd5-02d5f09fa2be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16971041-1519-4c28-9dd5-02d5f09fa2be" (UID: "16971041-1519-4c28-9dd5-02d5f09fa2be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 05:10:58.101679 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.101638 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16971041-1519-4c28-9dd5-02d5f09fa2be-tls-certs\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:58.101679 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.101675 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxsm\" (UniqueName: \"kubernetes.io/projected/16971041-1519-4c28-9dd5-02d5f09fa2be-kube-api-access-fhxsm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:58.101679 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.101684 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-dshm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:58.101679 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.101693 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16971041-1519-4c28-9dd5-02d5f09fa2be-model-cache\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:58.581133 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.581062 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" podUID="bc062c83-8be6-4677-831b-3a28bce6cd9d" containerName="main" containerID="cri-o://4df4102843fd35d189f99fc4d0bb9011aa1903474d8a8d83160c9fec81fd906b" gracePeriod=45 Jul 22 05:10:58.779990 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.779956 2572 generic.go:356] "Generic (PLEG): container finished" podID="16971041-1519-4c28-9dd5-02d5f09fa2be" containerID="de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a" exitCode=0 Jul 22 05:10:58.779990 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.779990 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" Jul 22 05:10:58.780490 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.780042 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" event={"ID":"16971041-1519-4c28-9dd5-02d5f09fa2be","Type":"ContainerDied","Data":"de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a"} Jul 22 05:10:58.780490 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.780091 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd" event={"ID":"16971041-1519-4c28-9dd5-02d5f09fa2be","Type":"ContainerDied","Data":"ccd7a3e30f66b28febc3e48b256ae5c707d9f2d47022cd589a5e46f4638723b0"} Jul 22 05:10:58.780490 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.780116 2572 scope.go:117] "RemoveContainer" containerID="de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a" Jul 22 05:10:58.781886 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.781860 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc062c83-8be6-4677-831b-3a28bce6cd9d" containerID="4df4102843fd35d189f99fc4d0bb9011aa1903474d8a8d83160c9fec81fd906b" exitCode=0 Jul 22 05:10:58.781995 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.781935 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" event={"ID":"bc062c83-8be6-4677-831b-3a28bce6cd9d","Type":"ContainerDied","Data":"4df4102843fd35d189f99fc4d0bb9011aa1903474d8a8d83160c9fec81fd906b"} Jul 22 05:10:58.790551 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.790533 2572 scope.go:117] "RemoveContainer" containerID="de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a" Jul 22 05:10:58.790798 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:10:58.790783 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a\": container with ID starting with de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a not found: ID does not exist" containerID="de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a" Jul 22 05:10:58.790842 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.790807 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a"} err="failed to get container status \"de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a\": rpc error: code = NotFound desc = could not find container \"de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a\": container with ID starting with de5359f209c45dc9707da972570f918aff03eeb532c4665a489963cd48e5a00a not found: ID does not exist" Jul 22 05:10:58.802090 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.802062 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f"] Jul 22 05:10:58.802414 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.802403 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fce892-qs84f"] Jul 22 05:10:58.804433 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.804413 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd"] Jul 22 05:10:58.811438 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.811417 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fce892/test-model-fce892-kserve-7484887587-gm8nd"] Jul 22 05:10:58.836539 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.836482 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:10:58.907451 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907419 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scjjh\" (UniqueName: \"kubernetes.io/projected/bc062c83-8be6-4677-831b-3a28bce6cd9d-kube-api-access-scjjh\") pod \"bc062c83-8be6-4677-831b-3a28bce6cd9d\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " Jul 22 05:10:58.907565 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907498 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-model-cache\") pod \"bc062c83-8be6-4677-831b-3a28bce6cd9d\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " Jul 22 05:10:58.907565 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907523 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc062c83-8be6-4677-831b-3a28bce6cd9d-tls-certs\") pod \"bc062c83-8be6-4677-831b-3a28bce6cd9d\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " Jul 22 05:10:58.907654 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907563 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-dshm\") pod \"bc062c83-8be6-4677-831b-3a28bce6cd9d\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " Jul 22 05:10:58.907654 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907599 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-home\") pod \"bc062c83-8be6-4677-831b-3a28bce6cd9d\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " Jul 22 05:10:58.907654 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907637 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-tmp-dir\") pod \"bc062c83-8be6-4677-831b-3a28bce6cd9d\" (UID: \"bc062c83-8be6-4677-831b-3a28bce6cd9d\") " Jul 22 05:10:58.907793 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907764 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc062c83-8be6-4677-831b-3a28bce6cd9d" (UID: "bc062c83-8be6-4677-831b-3a28bce6cd9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:10:58.907930 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907904 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-home" (OuterVolumeSpecName: "home") pod "bc062c83-8be6-4677-831b-3a28bce6cd9d" (UID: "bc062c83-8be6-4677-831b-3a28bce6cd9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:10:58.908044 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.907929 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-model-cache\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:58.908044 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.908035 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc062c83-8be6-4677-831b-3a28bce6cd9d" (UID: "bc062c83-8be6-4677-831b-3a28bce6cd9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:10:58.909750 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.909723 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-dshm" (OuterVolumeSpecName: "dshm") pod "bc062c83-8be6-4677-831b-3a28bce6cd9d" (UID: "bc062c83-8be6-4677-831b-3a28bce6cd9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:10:58.909836 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.909783 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc062c83-8be6-4677-831b-3a28bce6cd9d-kube-api-access-scjjh" (OuterVolumeSpecName: "kube-api-access-scjjh") pod "bc062c83-8be6-4677-831b-3a28bce6cd9d" (UID: "bc062c83-8be6-4677-831b-3a28bce6cd9d"). InnerVolumeSpecName "kube-api-access-scjjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:10:58.909836 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:58.909798 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc062c83-8be6-4677-831b-3a28bce6cd9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc062c83-8be6-4677-831b-3a28bce6cd9d" (UID: "bc062c83-8be6-4677-831b-3a28bce6cd9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 05:10:59.008442 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.008417 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc062c83-8be6-4677-831b-3a28bce6cd9d-tls-certs\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:59.008442 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.008437 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-dshm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:59.008442 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.008445 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-home\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:59.008652 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.008453 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc062c83-8be6-4677-831b-3a28bce6cd9d-tmp-dir\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:59.008652 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.008461 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scjjh\" (UniqueName: \"kubernetes.io/projected/bc062c83-8be6-4677-831b-3a28bce6cd9d-kube-api-access-scjjh\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:10:59.788775 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.788738 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" event={"ID":"bc062c83-8be6-4677-831b-3a28bce6cd9d","Type":"ContainerDied","Data":"77d0e7490ee423916b85effcb7307277edd662d03aebc9da665fc5ed8a2d65d0"} Jul 22 05:10:59.788775 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.788767 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4" Jul 22 05:10:59.789272 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.788796 2572 scope.go:117] "RemoveContainer" containerID="4df4102843fd35d189f99fc4d0bb9011aa1903474d8a8d83160c9fec81fd906b" Jul 22 05:10:59.808908 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.808882 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4"] Jul 22 05:10:59.812286 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.812264 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8e4aa9/test-model-8e4aa9-kserve-58b4c487f7-2wdc4"] Jul 22 05:10:59.960245 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:10:59.960221 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:11:00.800025 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:00.799996 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16971041-1519-4c28-9dd5-02d5f09fa2be" path="/var/lib/kubelet/pods/16971041-1519-4c28-9dd5-02d5f09fa2be/volumes" Jul 22 05:11:00.800404 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:00.800358 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc062c83-8be6-4677-831b-3a28bce6cd9d" path="/var/lib/kubelet/pods/bc062c83-8be6-4677-831b-3a28bce6cd9d/volumes" Jul 22 05:11:00.800672 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:00.800658 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918" path="/var/lib/kubelet/pods/d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918/volumes" Jul 22 05:11:05.488079 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488042 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg"] Jul 22 05:11:05.488497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488433 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc062c83-8be6-4677-831b-3a28bce6cd9d" containerName="main" Jul 22 05:11:05.488497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488445 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc062c83-8be6-4677-831b-3a28bce6cd9d" containerName="main" Jul 22 05:11:05.488497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488460 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918" containerName="revoke-keys" Jul 22 05:11:05.488497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488466 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918" containerName="revoke-keys" Jul 22 05:11:05.488497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488476 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16971041-1519-4c28-9dd5-02d5f09fa2be" containerName="main" Jul 22 05:11:05.488497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488481 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="16971041-1519-4c28-9dd5-02d5f09fa2be" containerName="main" Jul 22 05:11:05.488693 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488548 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="16971041-1519-4c28-9dd5-02d5f09fa2be" containerName="main" Jul 22 05:11:05.488693 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488558 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc062c83-8be6-4677-831b-3a28bce6cd9d" containerName="main" Jul 22 05:11:05.488693 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.488567 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7bc9c65-8bcb-4ad0-9f53-bfd0ec68b918" containerName="revoke-keys" Jul 22 05:11:05.493216 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.493200 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" Jul 22 05:11:05.495547 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.495523 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:11:05.495677 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.495531 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:11:05.502402 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.502362 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg"] Jul 22 05:11:05.567425 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.567363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15b65da-d643-45cc-84ef-1789a5d9d6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg\" (UID: \"e15b65da-d643-45cc-84ef-1789a5d9d6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" Jul 22 05:11:05.668256 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.668221 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15b65da-d643-45cc-84ef-1789a5d9d6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg\" (UID: \"e15b65da-d643-45cc-84ef-1789a5d9d6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" Jul 22 05:11:05.668897 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.668873 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15b65da-d643-45cc-84ef-1789a5d9d6b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg\" (UID: \"e15b65da-d643-45cc-84ef-1789a5d9d6b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" Jul 22 05:11:05.804112 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.804084 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" Jul 22 05:11:05.938156 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:05.938127 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg"] Jul 22 05:11:05.939576 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:11:05.939551 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15b65da_d643_45cc_84ef_1789a5d9d6b4.slice/crio-99b542a79efabf600ee4dc1dab80ac8def1e045457af0260f8dbbf9020386182 WatchSource:0}: Error finding container 99b542a79efabf600ee4dc1dab80ac8def1e045457af0260f8dbbf9020386182: Status 404 returned error can't find the container with id 99b542a79efabf600ee4dc1dab80ac8def1e045457af0260f8dbbf9020386182 Jul 22 05:11:06.828595 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:06.828561 2572 generic.go:356] "Generic (PLEG): container finished" podID="e15b65da-d643-45cc-84ef-1789a5d9d6b4" containerID="2f2c60c02b4c5593487616f5de9e30f681e86182d3de5a3abb76139981cf0f90" exitCode=0 Jul 22 05:11:06.828957 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:06.828596 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" event={"ID":"e15b65da-d643-45cc-84ef-1789a5d9d6b4","Type":"ContainerDied","Data":"2f2c60c02b4c5593487616f5de9e30f681e86182d3de5a3abb76139981cf0f90"} Jul 22 05:11:06.828957 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:06.828649 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" event={"ID":"e15b65da-d643-45cc-84ef-1789a5d9d6b4","Type":"ContainerStarted","Data":"99b542a79efabf600ee4dc1dab80ac8def1e045457af0260f8dbbf9020386182"} Jul 22 05:11:07.989977 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:07.989950 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" Jul 22 05:11:08.090271 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:08.090165 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15b65da-d643-45cc-84ef-1789a5d9d6b4-maas-api-service-ca\") pod \"e15b65da-d643-45cc-84ef-1789a5d9d6b4\" (UID: \"e15b65da-d643-45cc-84ef-1789a5d9d6b4\") " Jul 22 05:11:08.090550 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:08.090527 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15b65da-d643-45cc-84ef-1789a5d9d6b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e15b65da-d643-45cc-84ef-1789a5d9d6b4" (UID: "e15b65da-d643-45cc-84ef-1789a5d9d6b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:11:08.191281 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:08.191242 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e15b65da-d643-45cc-84ef-1789a5d9d6b4-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:11:08.841510 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:08.841481 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" Jul 22 05:11:08.841655 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:08.841507 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg" event={"ID":"e15b65da-d643-45cc-84ef-1789a5d9d6b4","Type":"ContainerDied","Data":"99b542a79efabf600ee4dc1dab80ac8def1e045457af0260f8dbbf9020386182"} Jul 22 05:11:08.841655 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:08.841550 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99b542a79efabf600ee4dc1dab80ac8def1e045457af0260f8dbbf9020386182" Jul 22 05:11:19.053127 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:19.053090 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:11:20.568431 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:20.568387 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg"] Jul 22 05:11:20.575754 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:20.575721 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8e4aa9-hrbwg"] Jul 22 05:11:20.801435 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:20.801393 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e15b65da-d643-45cc-84ef-1789a5d9d6b4" path="/var/lib/kubelet/pods/e15b65da-d643-45cc-84ef-1789a5d9d6b4/volumes" Jul 22 05:11:36.468875 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:36.468842 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:11:52.137487 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.137449 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644"] Jul 22 05:11:52.137960 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.137809 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e15b65da-d643-45cc-84ef-1789a5d9d6b4" containerName="revoke-keys" Jul 22 05:11:52.137960 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.137820 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e15b65da-d643-45cc-84ef-1789a5d9d6b4" containerName="revoke-keys" Jul 22 05:11:52.137960 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.137883 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e15b65da-d643-45cc-84ef-1789a5d9d6b4" containerName="revoke-keys" Jul 22 05:11:52.140998 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.140980 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" Jul 22 05:11:52.143265 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.143242 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:11:52.143265 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.143259 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:11:52.147590 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.147560 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644"] Jul 22 05:11:52.153110 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.153084 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac873e3-b75c-4790-93a1-53ef9d8a70a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3de18-ww644\" (UID: \"bac873e3-b75c-4790-93a1-53ef9d8a70a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" Jul 22 05:11:52.254387 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.254341 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac873e3-b75c-4790-93a1-53ef9d8a70a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3de18-ww644\" (UID: \"bac873e3-b75c-4790-93a1-53ef9d8a70a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" Jul 22 05:11:52.254933 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.254913 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac873e3-b75c-4790-93a1-53ef9d8a70a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3de18-ww644\" (UID: \"bac873e3-b75c-4790-93a1-53ef9d8a70a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" Jul 22 05:11:52.453523 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.453442 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" Jul 22 05:11:52.585425 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:52.585401 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644"] Jul 22 05:11:52.587921 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:11:52.587891 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac873e3_b75c_4790_93a1_53ef9d8a70a9.slice/crio-040f0780fb4d744004ed5832d81f6789bb4add65160454386e19bd4ac2b948a8 WatchSource:0}: Error finding container 040f0780fb4d744004ed5832d81f6789bb4add65160454386e19bd4ac2b948a8: Status 404 returned error can't find the container with id 040f0780fb4d744004ed5832d81f6789bb4add65160454386e19bd4ac2b948a8 Jul 22 05:11:53.074954 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:53.074918 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" event={"ID":"bac873e3-b75c-4790-93a1-53ef9d8a70a9","Type":"ContainerStarted","Data":"e1a84d29da593e1dc9211a954bd07007b19ac859a8c70e166646bf0b0a9d7b13"} Jul 22 05:11:53.075141 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:53.074987 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" event={"ID":"bac873e3-b75c-4790-93a1-53ef9d8a70a9","Type":"ContainerStarted","Data":"040f0780fb4d744004ed5832d81f6789bb4add65160454386e19bd4ac2b948a8"} Jul 22 05:11:53.091339 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:53.091290 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" podStartSLOduration=1.091276246 podStartE2EDuration="1.091276246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:11:53.088220498 +0000 UTC m=+2324.806825644" watchObservedRunningTime="2026-07-22 05:11:53.091276246 +0000 UTC m=+2324.809881380" Jul 22 05:11:53.758309 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:53.758273 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:11:54.088145 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:54.087961 2572 generic.go:356] "Generic (PLEG): container finished" podID="bac873e3-b75c-4790-93a1-53ef9d8a70a9" containerID="e1a84d29da593e1dc9211a954bd07007b19ac859a8c70e166646bf0b0a9d7b13" exitCode=0 Jul 22 05:11:54.088296 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:54.088110 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" event={"ID":"bac873e3-b75c-4790-93a1-53ef9d8a70a9","Type":"ContainerDied","Data":"e1a84d29da593e1dc9211a954bd07007b19ac859a8c70e166646bf0b0a9d7b13"} Jul 22 05:11:55.229521 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:55.229495 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" Jul 22 05:11:55.282886 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:55.282856 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac873e3-b75c-4790-93a1-53ef9d8a70a9-maas-api-service-ca\") pod \"bac873e3-b75c-4790-93a1-53ef9d8a70a9\" (UID: \"bac873e3-b75c-4790-93a1-53ef9d8a70a9\") " Jul 22 05:11:55.283212 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:55.283192 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac873e3-b75c-4790-93a1-53ef9d8a70a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bac873e3-b75c-4790-93a1-53ef9d8a70a9" (UID: "bac873e3-b75c-4790-93a1-53ef9d8a70a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:11:55.383815 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:55.383746 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac873e3-b75c-4790-93a1-53ef9d8a70a9-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:11:56.101126 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:56.101096 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" Jul 22 05:11:56.101301 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:56.101123 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644" event={"ID":"bac873e3-b75c-4790-93a1-53ef9d8a70a9","Type":"ContainerDied","Data":"040f0780fb4d744004ed5832d81f6789bb4add65160454386e19bd4ac2b948a8"} Jul 22 05:11:56.101301 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:11:56.101168 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="040f0780fb4d744004ed5832d81f6789bb4add65160454386e19bd4ac2b948a8" Jul 22 05:12:09.043341 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:09.043269 2572 scope.go:117] "RemoveContainer" containerID="8e1523e4a556e4acd4af88921014707391182afc8108182fe4b126cae8d2b5e7" Jul 22 05:12:11.859014 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:11.858978 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:12:12.222352 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:12.222269 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644"] Jul 22 05:12:12.224414 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:12.224367 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3de18-ww644"] Jul 22 05:12:12.800690 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:12.800650 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac873e3-b75c-4790-93a1-53ef9d8a70a9" path="/var/lib/kubelet/pods/bac873e3-b75c-4790-93a1-53ef9d8a70a9/volumes" Jul 22 05:12:18.950643 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:18.950611 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv"] Jul 22 05:12:18.951006 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:18.950947 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac873e3-b75c-4790-93a1-53ef9d8a70a9" containerName="revoke-keys" Jul 22 05:12:18.951006 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:18.950957 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac873e3-b75c-4790-93a1-53ef9d8a70a9" containerName="revoke-keys" Jul 22 05:12:18.951077 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:18.951020 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac873e3-b75c-4790-93a1-53ef9d8a70a9" containerName="revoke-keys" Jul 22 05:12:18.959327 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:18.959296 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" Jul 22 05:12:18.961599 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:18.961561 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv"] Jul 22 05:12:18.961892 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:18.961854 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:12:18.962022 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:18.962003 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:12:19.087905 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:19.087873 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce2560f-5cd3-4ef5-b479-a9d397e78b86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv\" (UID: \"fce2560f-5cd3-4ef5-b479-a9d397e78b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" Jul 22 05:12:19.189093 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:19.189054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce2560f-5cd3-4ef5-b479-a9d397e78b86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv\" (UID: \"fce2560f-5cd3-4ef5-b479-a9d397e78b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" Jul 22 05:12:19.189716 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:19.189690 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce2560f-5cd3-4ef5-b479-a9d397e78b86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv\" (UID: \"fce2560f-5cd3-4ef5-b479-a9d397e78b86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" Jul 22 05:12:19.270688 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:19.270663 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" Jul 22 05:12:19.402688 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:19.402660 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv"] Jul 22 05:12:19.405364 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:12:19.405322 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce2560f_5cd3_4ef5_b479_a9d397e78b86.slice/crio-d922f4edffda17a9cc91617e3802aeab6d8def03952a0b64343f8880a61613e6 WatchSource:0}: Error finding container d922f4edffda17a9cc91617e3802aeab6d8def03952a0b64343f8880a61613e6: Status 404 returned error can't find the container with id d922f4edffda17a9cc91617e3802aeab6d8def03952a0b64343f8880a61613e6 Jul 22 05:12:20.234745 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:20.234707 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" event={"ID":"fce2560f-5cd3-4ef5-b479-a9d397e78b86","Type":"ContainerStarted","Data":"55e1efad024747ea89b1dd7985e6b58e15eb4866976672b866ba3fdb51744d47"} Jul 22 05:12:20.235123 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:20.234779 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" event={"ID":"fce2560f-5cd3-4ef5-b479-a9d397e78b86","Type":"ContainerStarted","Data":"d922f4edffda17a9cc91617e3802aeab6d8def03952a0b64343f8880a61613e6"} Jul 22 05:12:20.249965 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:20.249918 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" podStartSLOduration=2.249902107 podStartE2EDuration="2.249902107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:12:20.248422598 +0000 UTC m=+2351.967027733" watchObservedRunningTime="2026-07-22 05:12:20.249902107 +0000 UTC m=+2351.968507267" Jul 22 05:12:21.241254 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:21.241221 2572 generic.go:356] "Generic (PLEG): container finished" podID="fce2560f-5cd3-4ef5-b479-a9d397e78b86" containerID="55e1efad024747ea89b1dd7985e6b58e15eb4866976672b866ba3fdb51744d47" exitCode=0 Jul 22 05:12:21.241675 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:21.241314 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" event={"ID":"fce2560f-5cd3-4ef5-b479-a9d397e78b86","Type":"ContainerDied","Data":"55e1efad024747ea89b1dd7985e6b58e15eb4866976672b866ba3fdb51744d47"} Jul 22 05:12:22.377065 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:22.377038 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" Jul 22 05:12:22.516579 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:22.516551 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce2560f-5cd3-4ef5-b479-a9d397e78b86-maas-api-service-ca\") pod \"fce2560f-5cd3-4ef5-b479-a9d397e78b86\" (UID: \"fce2560f-5cd3-4ef5-b479-a9d397e78b86\") " Jul 22 05:12:22.516887 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:22.516866 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce2560f-5cd3-4ef5-b479-a9d397e78b86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fce2560f-5cd3-4ef5-b479-a9d397e78b86" (UID: "fce2560f-5cd3-4ef5-b479-a9d397e78b86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:12:22.618143 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:22.618113 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce2560f-5cd3-4ef5-b479-a9d397e78b86-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:12:23.252479 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:23.252397 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" Jul 22 05:12:23.252479 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:23.252423 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv" event={"ID":"fce2560f-5cd3-4ef5-b479-a9d397e78b86","Type":"ContainerDied","Data":"d922f4edffda17a9cc91617e3802aeab6d8def03952a0b64343f8880a61613e6"} Jul 22 05:12:23.252479 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:23.252469 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d922f4edffda17a9cc91617e3802aeab6d8def03952a0b64343f8880a61613e6" Jul 22 05:12:39.025630 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:39.025593 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv"] Jul 22 05:12:39.028692 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:39.028669 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-04dfa1-9qzjv"] Jul 22 05:12:39.260253 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:39.260156 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:12:40.800714 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:40.800683 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce2560f-5cd3-4ef5-b479-a9d397e78b86" path="/var/lib/kubelet/pods/fce2560f-5cd3-4ef5-b479-a9d397e78b86/volumes" Jul 22 05:12:57.360484 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:12:57.360445 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:13:08.953262 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:08.953234 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:13:08.959873 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:08.959847 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:13:12.370723 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:12.370685 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:13:17.466102 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.466068 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb"] Jul 22 05:13:17.466785 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.466457 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce2560f-5cd3-4ef5-b479-a9d397e78b86" containerName="revoke-keys" Jul 22 05:13:17.466785 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.466469 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce2560f-5cd3-4ef5-b479-a9d397e78b86" containerName="revoke-keys" Jul 22 05:13:17.466785 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.466534 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce2560f-5cd3-4ef5-b479-a9d397e78b86" containerName="revoke-keys" Jul 22 05:13:17.469607 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.469587 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.471679 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.471652 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c0234d\"/\"default-dockercfg-tg6lh\"" Jul 22 05:13:17.472607 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.472586 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c0234d\"/\"auth-test-model-c0234d-kserve-self-signed-certs\"" Jul 22 05:13:17.472702 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.472636 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c0234d\"/\"openshift-service-ca.crt\"" Jul 22 05:13:17.472860 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.472833 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c0234d\"/\"kube-root-ca.crt\"" Jul 22 05:13:17.478702 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.478676 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb"] Jul 22 05:13:17.576053 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.576023 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-tmp-dir\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.576208 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.576060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-home\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.576208 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.576077 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-dshm\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.576208 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.576102 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx9r4\" (UniqueName: \"kubernetes.io/projected/8301ceb7-def3-4e75-b283-ebdeaca9a877-kube-api-access-sx9r4\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.576208 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.576197 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8301ceb7-def3-4e75-b283-ebdeaca9a877-tls-certs\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.576348 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.576230 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-model-cache\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677146 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8301ceb7-def3-4e75-b283-ebdeaca9a877-tls-certs\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677316 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677157 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-model-cache\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677316 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677180 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-tmp-dir\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677316 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677205 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-home\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677316 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-dshm\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677551 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677339 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx9r4\" (UniqueName: \"kubernetes.io/projected/8301ceb7-def3-4e75-b283-ebdeaca9a877-kube-api-access-sx9r4\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677633 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677614 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-model-cache\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677700 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677642 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-tmp-dir\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.677757 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.677697 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-home\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.679508 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.679485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-dshm\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.679763 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.679745 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8301ceb7-def3-4e75-b283-ebdeaca9a877-tls-certs\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.684454 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.684438 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx9r4\" (UniqueName: \"kubernetes.io/projected/8301ceb7-def3-4e75-b283-ebdeaca9a877-kube-api-access-sx9r4\") pod \"auth-test-model-c0234d-kserve-7bc8d665cf-2dntb\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.780595 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.780569 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:17.907954 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:17.907921 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb"] Jul 22 05:13:17.909489 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:13:17.909461 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8301ceb7_def3_4e75_b283_ebdeaca9a877.slice/crio-ac36b77011cb63f516c236ae89c84e83e5aaeaf1d7009f34f2322d4c2e48ba8b WatchSource:0}: Error finding container ac36b77011cb63f516c236ae89c84e83e5aaeaf1d7009f34f2322d4c2e48ba8b: Status 404 returned error can't find the container with id ac36b77011cb63f516c236ae89c84e83e5aaeaf1d7009f34f2322d4c2e48ba8b Jul 22 05:13:18.551473 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:18.551434 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" event={"ID":"8301ceb7-def3-4e75-b283-ebdeaca9a877","Type":"ContainerStarted","Data":"56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0"} Jul 22 05:13:18.551910 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:18.551481 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" event={"ID":"8301ceb7-def3-4e75-b283-ebdeaca9a877","Type":"ContainerStarted","Data":"ac36b77011cb63f516c236ae89c84e83e5aaeaf1d7009f34f2322d4c2e48ba8b"} Jul 22 05:13:18.566974 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:18.566917 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" podStartSLOduration=1.566897725 podStartE2EDuration="1.566897725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:13:18.565837847 +0000 UTC m=+2410.284442982" watchObservedRunningTime="2026-07-22 05:13:18.566897725 +0000 UTC m=+2410.285502862" Jul 22 05:13:22.752161 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:22.752122 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:13:27.781663 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:27.781631 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:27.782145 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:27.781691 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:27.794186 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:27.794156 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:28.622114 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:28.622081 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:13:44.261590 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.261550 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d"] Jul 22 05:13:44.265504 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.265482 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.267677 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.267654 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1a3bbb\"/\"kube-root-ca.crt\"" Jul 22 05:13:44.268593 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.268575 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1a3bbb\"/\"auth-test-model-1a3bbb-kserve-self-signed-certs\"" Jul 22 05:13:44.268680 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.268606 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1a3bbb\"/\"openshift-service-ca.crt\"" Jul 22 05:13:44.268680 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.268614 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1a3bbb\"/\"default-dockercfg-dmxqm\"" Jul 22 05:13:44.272842 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.272817 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d"] Jul 22 05:13:44.289885 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.289861 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-tmp-dir\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.289988 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.289895 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-home\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.289988 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.289917 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-dshm\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.289988 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.289933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-model-cache\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.289988 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.289948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f4464-96a5-4e1f-801b-345b81804c4b-tls-certs\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.290154 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.290105 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xv9f\" (UniqueName: \"kubernetes.io/projected/0e8f4464-96a5-4e1f-801b-345b81804c4b-kube-api-access-6xv9f\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.391610 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.391570 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-tmp-dir\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.391799 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.391619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-home\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.391799 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.391649 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-dshm\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.391799 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.391673 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-model-cache\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.391799 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.391694 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f4464-96a5-4e1f-801b-345b81804c4b-tls-certs\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.391799 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.391766 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xv9f\" (UniqueName: \"kubernetes.io/projected/0e8f4464-96a5-4e1f-801b-345b81804c4b-kube-api-access-6xv9f\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.392054 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.392006 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-tmp-dir\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.392054 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.392023 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-model-cache\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.392318 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.392296 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-home\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.394077 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.394045 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-dshm\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.394213 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.394190 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f4464-96a5-4e1f-801b-345b81804c4b-tls-certs\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.398967 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.398941 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xv9f\" (UniqueName: \"kubernetes.io/projected/0e8f4464-96a5-4e1f-801b-345b81804c4b-kube-api-access-6xv9f\") pod \"auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.576437 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.576401 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:44.706883 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:44.706856 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d"] Jul 22 05:13:44.707908 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:13:44.707881 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e8f4464_96a5_4e1f_801b_345b81804c4b.slice/crio-73d7a01f24f40acf131267d724933e9036c7b3683686e2ce8bb4efba89ea7624 WatchSource:0}: Error finding container 73d7a01f24f40acf131267d724933e9036c7b3683686e2ce8bb4efba89ea7624: Status 404 returned error can't find the container with id 73d7a01f24f40acf131267d724933e9036c7b3683686e2ce8bb4efba89ea7624 Jul 22 05:13:45.710193 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:45.710159 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" event={"ID":"0e8f4464-96a5-4e1f-801b-345b81804c4b","Type":"ContainerStarted","Data":"bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6"} Jul 22 05:13:45.710623 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:45.710201 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" event={"ID":"0e8f4464-96a5-4e1f-801b-345b81804c4b","Type":"ContainerStarted","Data":"73d7a01f24f40acf131267d724933e9036c7b3683686e2ce8bb4efba89ea7624"} Jul 22 05:13:45.726328 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:45.726280 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" podStartSLOduration=1.726265765 podStartE2EDuration="1.726265765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:13:45.724185811 +0000 UTC m=+2437.442790947" watchObservedRunningTime="2026-07-22 05:13:45.726265765 +0000 UTC m=+2437.444870899" Jul 22 05:13:47.762300 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:47.762263 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:13:54.577108 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:54.577074 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:54.577509 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:54.577120 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:54.589469 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:54.589448 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:13:54.771216 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:13:54.771183 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:14:15.659629 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:14:15.659592 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:14:33.055466 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:14:33.055429 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:14:49.954403 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:14:49.954356 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:15:00.136765 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.136730 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh"] Jul 22 05:15:00.140500 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.140478 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7"] Jul 22 05:15:00.140655 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.140618 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" Jul 22 05:15:00.142755 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.142737 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:15:00.143633 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.143618 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" Jul 22 05:15:00.149544 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.149524 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh"] Jul 22 05:15:00.153284 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.153264 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" Jul 22 05:15:00.158163 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.157941 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7"] Jul 22 05:15:00.162696 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.162680 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" Jul 22 05:15:00.306315 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.305269 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh"] Jul 22 05:15:00.306849 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:15:00.306813 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8d3b73_9ee3_40e2_8fdf_4277435fd63e.slice/crio-b95a037fc21b2e77e2fb94d78a137755781b083fbf705e2bf8d300526876c369 WatchSource:0}: Error finding container b95a037fc21b2e77e2fb94d78a137755781b083fbf705e2bf8d300526876c369: Status 404 returned error can't find the container with id b95a037fc21b2e77e2fb94d78a137755781b083fbf705e2bf8d300526876c369 Jul 22 05:15:00.332360 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:00.332332 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7"] Jul 22 05:15:00.333551 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:15:00.333528 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c84346_729d_4da9_83f1_174ad9d09561.slice/crio-aa83afccebe6dc9dcfd4a7d3d754779726119c8bb883d37c6fad9cf4163f8d56 WatchSource:0}: Error finding container aa83afccebe6dc9dcfd4a7d3d754779726119c8bb883d37c6fad9cf4163f8d56: Status 404 returned error can't find the container with id aa83afccebe6dc9dcfd4a7d3d754779726119c8bb883d37c6fad9cf4163f8d56 Jul 22 05:15:01.127741 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:01.127706 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" event={"ID":"ff8d3b73-9ee3-40e2-8fdf-4277435fd63e","Type":"ContainerStarted","Data":"1fce60e8072b4ff70f50206a0d9b1a6235362520bf6b15edf0424cc586d56416"} Jul 22 05:15:01.127937 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:01.127757 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" event={"ID":"ff8d3b73-9ee3-40e2-8fdf-4277435fd63e","Type":"ContainerStarted","Data":"b95a037fc21b2e77e2fb94d78a137755781b083fbf705e2bf8d300526876c369"} Jul 22 05:15:01.129332 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:01.129302 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" event={"ID":"75c84346-729d-4da9-83f1-174ad9d09561","Type":"ContainerStarted","Data":"1838ef78969131367d2292fba09bd319f2b5caf9998bbb38f465ef9df2c24fb0"} Jul 22 05:15:01.129497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:01.129346 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" event={"ID":"75c84346-729d-4da9-83f1-174ad9d09561","Type":"ContainerStarted","Data":"aa83afccebe6dc9dcfd4a7d3d754779726119c8bb883d37c6fad9cf4163f8d56"} Jul 22 05:15:01.141879 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:01.141838 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" podStartSLOduration=1.141824153 podStartE2EDuration="1.141824153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:15:01.140908024 +0000 UTC m=+2512.859513161" watchObservedRunningTime="2026-07-22 05:15:01.141824153 +0000 UTC m=+2512.860429288" Jul 22 05:15:01.154192 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:01.154149 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" podStartSLOduration=1.15413513 podStartE2EDuration="1.15413513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:15:01.152922617 +0000 UTC m=+2512.871527751" watchObservedRunningTime="2026-07-22 05:15:01.15413513 +0000 UTC m=+2512.872740265" Jul 22 05:15:02.142180 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:02.142146 2572 generic.go:356] "Generic (PLEG): container finished" podID="ff8d3b73-9ee3-40e2-8fdf-4277435fd63e" containerID="1fce60e8072b4ff70f50206a0d9b1a6235362520bf6b15edf0424cc586d56416" exitCode=0 Jul 22 05:15:02.142680 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:02.142239 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" event={"ID":"ff8d3b73-9ee3-40e2-8fdf-4277435fd63e","Type":"ContainerDied","Data":"1fce60e8072b4ff70f50206a0d9b1a6235362520bf6b15edf0424cc586d56416"} Jul 22 05:15:02.143707 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:02.143685 2572 generic.go:356] "Generic (PLEG): container finished" podID="75c84346-729d-4da9-83f1-174ad9d09561" containerID="1838ef78969131367d2292fba09bd319f2b5caf9998bbb38f465ef9df2c24fb0" exitCode=0 Jul 22 05:15:02.143826 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:02.143732 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" event={"ID":"75c84346-729d-4da9-83f1-174ad9d09561","Type":"ContainerDied","Data":"1838ef78969131367d2292fba09bd319f2b5caf9998bbb38f465ef9df2c24fb0"} Jul 22 05:15:03.304535 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:03.304500 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" Jul 22 05:15:03.307264 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:03.307247 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" Jul 22 05:15:04.156715 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:04.156685 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" Jul 22 05:15:04.156715 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:04.156694 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh" event={"ID":"ff8d3b73-9ee3-40e2-8fdf-4277435fd63e","Type":"ContainerDied","Data":"b95a037fc21b2e77e2fb94d78a137755781b083fbf705e2bf8d300526876c369"} Jul 22 05:15:04.157082 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:04.156737 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b95a037fc21b2e77e2fb94d78a137755781b083fbf705e2bf8d300526876c369" Jul 22 05:15:04.157955 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:04.157934 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" Jul 22 05:15:04.158074 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:04.157936 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7" event={"ID":"75c84346-729d-4da9-83f1-174ad9d09561","Type":"ContainerDied","Data":"aa83afccebe6dc9dcfd4a7d3d754779726119c8bb883d37c6fad9cf4163f8d56"} Jul 22 05:15:04.158074 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:04.158065 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa83afccebe6dc9dcfd4a7d3d754779726119c8bb883d37c6fad9cf4163f8d56" Jul 22 05:15:08.259043 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:08.259010 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:15:22.858477 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:22.858443 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:15:36.361420 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:36.361370 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:15:52.963834 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:52.963794 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:15:58.140158 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:58.140121 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb"] Jul 22 05:15:59.073025 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:15:59.072987 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d"] Jul 22 05:16:10.958701 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:10.958662 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:16:13.160534 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.160480 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" podUID="8301ceb7-def3-4e75-b283-ebdeaca9a877" containerName="main" containerID="cri-o://56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0" gracePeriod=45 Jul 22 05:16:13.409746 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.409721 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:16:13.528676 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.528645 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8301ceb7-def3-4e75-b283-ebdeaca9a877-tls-certs\") pod \"8301ceb7-def3-4e75-b283-ebdeaca9a877\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " Jul 22 05:16:13.528831 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.528728 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-tmp-dir\") pod \"8301ceb7-def3-4e75-b283-ebdeaca9a877\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " Jul 22 05:16:13.528831 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.528776 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sx9r4\" (UniqueName: \"kubernetes.io/projected/8301ceb7-def3-4e75-b283-ebdeaca9a877-kube-api-access-sx9r4\") pod \"8301ceb7-def3-4e75-b283-ebdeaca9a877\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " Jul 22 05:16:13.528831 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.528792 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-home\") pod \"8301ceb7-def3-4e75-b283-ebdeaca9a877\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " Jul 22 05:16:13.528831 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.528813 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-dshm\") pod \"8301ceb7-def3-4e75-b283-ebdeaca9a877\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " Jul 22 05:16:13.529008 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.528872 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-model-cache\") pod \"8301ceb7-def3-4e75-b283-ebdeaca9a877\" (UID: \"8301ceb7-def3-4e75-b283-ebdeaca9a877\") " Jul 22 05:16:13.529064 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.528998 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8301ceb7-def3-4e75-b283-ebdeaca9a877" (UID: "8301ceb7-def3-4e75-b283-ebdeaca9a877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:16:13.529127 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.529072 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-home" (OuterVolumeSpecName: "home") pod "8301ceb7-def3-4e75-b283-ebdeaca9a877" (UID: "8301ceb7-def3-4e75-b283-ebdeaca9a877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:16:13.529214 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.529198 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-tmp-dir\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:13.529278 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.529215 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-home\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:13.529278 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.529220 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-model-cache" (OuterVolumeSpecName: "model-cache") pod "8301ceb7-def3-4e75-b283-ebdeaca9a877" (UID: "8301ceb7-def3-4e75-b283-ebdeaca9a877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:16:13.531050 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.531022 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8301ceb7-def3-4e75-b283-ebdeaca9a877-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8301ceb7-def3-4e75-b283-ebdeaca9a877" (UID: "8301ceb7-def3-4e75-b283-ebdeaca9a877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 05:16:13.531153 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.531045 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8301ceb7-def3-4e75-b283-ebdeaca9a877-kube-api-access-sx9r4" (OuterVolumeSpecName: "kube-api-access-sx9r4") pod "8301ceb7-def3-4e75-b283-ebdeaca9a877" (UID: "8301ceb7-def3-4e75-b283-ebdeaca9a877"). InnerVolumeSpecName "kube-api-access-sx9r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:16:13.531153 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.531022 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-dshm" (OuterVolumeSpecName: "dshm") pod "8301ceb7-def3-4e75-b283-ebdeaca9a877" (UID: "8301ceb7-def3-4e75-b283-ebdeaca9a877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:16:13.539490 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.539466 2572 generic.go:356] "Generic (PLEG): container finished" podID="8301ceb7-def3-4e75-b283-ebdeaca9a877" containerID="56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0" exitCode=0 Jul 22 05:16:13.539612 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.539526 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" Jul 22 05:16:13.539612 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.539557 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" event={"ID":"8301ceb7-def3-4e75-b283-ebdeaca9a877","Type":"ContainerDied","Data":"56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0"} Jul 22 05:16:13.539715 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.539623 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb" event={"ID":"8301ceb7-def3-4e75-b283-ebdeaca9a877","Type":"ContainerDied","Data":"ac36b77011cb63f516c236ae89c84e83e5aaeaf1d7009f34f2322d4c2e48ba8b"} Jul 22 05:16:13.539715 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.539648 2572 scope.go:117] "RemoveContainer" containerID="56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0" Jul 22 05:16:13.550988 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.550973 2572 scope.go:117] "RemoveContainer" containerID="56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0" Jul 22 05:16:13.551240 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:16:13.551221 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0\": container with ID starting with 56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0 not found: ID does not exist" containerID="56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0" Jul 22 05:16:13.551286 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.551250 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0"} err="failed to get container status \"56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0\": rpc error: code = NotFound desc = could not find container \"56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0\": container with ID starting with 56ee2188d4d9d0f233ac92203c269f96f31bcab8404c6760a78f043842a21da0 not found: ID does not exist" Jul 22 05:16:13.561411 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.561388 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb"] Jul 22 05:16:13.567065 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.567046 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/auth-test-model-c0234d-kserve-7bc8d665cf-2dntb"] Jul 22 05:16:13.630553 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.630526 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-model-cache\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:13.630553 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.630550 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8301ceb7-def3-4e75-b283-ebdeaca9a877-tls-certs\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:13.630685 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.630558 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx9r4\" (UniqueName: \"kubernetes.io/projected/8301ceb7-def3-4e75-b283-ebdeaca9a877-kube-api-access-sx9r4\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:13.630685 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:13.630568 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8301ceb7-def3-4e75-b283-ebdeaca9a877-dshm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:14.094304 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.094268 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" podUID="0e8f4464-96a5-4e1f-801b-345b81804c4b" containerName="main" containerID="cri-o://bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6" gracePeriod=45 Jul 22 05:16:14.363963 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.363936 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:16:14.436070 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436036 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f4464-96a5-4e1f-801b-345b81804c4b-tls-certs\") pod \"0e8f4464-96a5-4e1f-801b-345b81804c4b\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " Jul 22 05:16:14.436250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436133 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-home\") pod \"0e8f4464-96a5-4e1f-801b-345b81804c4b\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " Jul 22 05:16:14.436250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436151 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xv9f\" (UniqueName: \"kubernetes.io/projected/0e8f4464-96a5-4e1f-801b-345b81804c4b-kube-api-access-6xv9f\") pod \"0e8f4464-96a5-4e1f-801b-345b81804c4b\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " Jul 22 05:16:14.436250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436196 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-dshm\") pod \"0e8f4464-96a5-4e1f-801b-345b81804c4b\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " Jul 22 05:16:14.436250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436226 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-model-cache\") pod \"0e8f4464-96a5-4e1f-801b-345b81804c4b\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " Jul 22 05:16:14.436250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436250 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-tmp-dir\") pod \"0e8f4464-96a5-4e1f-801b-345b81804c4b\" (UID: \"0e8f4464-96a5-4e1f-801b-345b81804c4b\") " Jul 22 05:16:14.436573 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436337 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-home" (OuterVolumeSpecName: "home") pod "0e8f4464-96a5-4e1f-801b-345b81804c4b" (UID: "0e8f4464-96a5-4e1f-801b-345b81804c4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:16:14.436573 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436512 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-home\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:14.436573 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436544 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e8f4464-96a5-4e1f-801b-345b81804c4b" (UID: "0e8f4464-96a5-4e1f-801b-345b81804c4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:16:14.436684 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.436588 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e8f4464-96a5-4e1f-801b-345b81804c4b" (UID: "0e8f4464-96a5-4e1f-801b-345b81804c4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:16:14.438314 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.438292 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-dshm" (OuterVolumeSpecName: "dshm") pod "0e8f4464-96a5-4e1f-801b-345b81804c4b" (UID: "0e8f4464-96a5-4e1f-801b-345b81804c4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:16:14.438438 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.438328 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8f4464-96a5-4e1f-801b-345b81804c4b-kube-api-access-6xv9f" (OuterVolumeSpecName: "kube-api-access-6xv9f") pod "0e8f4464-96a5-4e1f-801b-345b81804c4b" (UID: "0e8f4464-96a5-4e1f-801b-345b81804c4b"). InnerVolumeSpecName "kube-api-access-6xv9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:16:14.438438 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.438360 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8f4464-96a5-4e1f-801b-345b81804c4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e8f4464-96a5-4e1f-801b-345b81804c4b" (UID: "0e8f4464-96a5-4e1f-801b-345b81804c4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 05:16:14.537725 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.537704 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-dshm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:14.537725 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.537726 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-model-cache\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:14.537913 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.537736 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8f4464-96a5-4e1f-801b-345b81804c4b-tmp-dir\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:14.537913 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.537744 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f4464-96a5-4e1f-801b-345b81804c4b-tls-certs\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:14.537913 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.537753 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xv9f\" (UniqueName: \"kubernetes.io/projected/0e8f4464-96a5-4e1f-801b-345b81804c4b-kube-api-access-6xv9f\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:16:14.547321 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.547292 2572 generic.go:356] "Generic (PLEG): container finished" podID="0e8f4464-96a5-4e1f-801b-345b81804c4b" containerID="bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6" exitCode=0 Jul 22 05:16:14.547440 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.547333 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" Jul 22 05:16:14.547440 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.547394 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" event={"ID":"0e8f4464-96a5-4e1f-801b-345b81804c4b","Type":"ContainerDied","Data":"bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6"} Jul 22 05:16:14.547440 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.547438 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d" event={"ID":"0e8f4464-96a5-4e1f-801b-345b81804c4b","Type":"ContainerDied","Data":"73d7a01f24f40acf131267d724933e9036c7b3683686e2ce8bb4efba89ea7624"} Jul 22 05:16:14.547600 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.547457 2572 scope.go:117] "RemoveContainer" containerID="bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6" Jul 22 05:16:14.556508 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.556489 2572 scope.go:117] "RemoveContainer" containerID="bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6" Jul 22 05:16:14.556747 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:16:14.556731 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6\": container with ID starting with bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6 not found: ID does not exist" containerID="bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6" Jul 22 05:16:14.556791 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.556753 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6"} err="failed to get container status \"bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6\": rpc error: code = NotFound desc = could not find container \"bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6\": container with ID starting with bf67e7af9217de0b613c95b17e005585bede088e0bff0b00cf0f8b33432944f6 not found: ID does not exist" Jul 22 05:16:14.568599 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.568567 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d"] Jul 22 05:16:14.570795 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.570762 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/auth-test-model-1a3bbb-kserve-6dcd596ddc-6686d"] Jul 22 05:16:14.800798 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.800771 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e8f4464-96a5-4e1f-801b-345b81804c4b" path="/var/lib/kubelet/pods/0e8f4464-96a5-4e1f-801b-345b81804c4b/volumes" Jul 22 05:16:14.801201 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:14.801186 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8301ceb7-def3-4e75-b283-ebdeaca9a877" path="/var/lib/kubelet/pods/8301ceb7-def3-4e75-b283-ebdeaca9a877/volumes" Jul 22 05:16:21.353408 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:21.353358 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:16:48.067723 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.067685 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk"] Jul 22 05:16:48.068307 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068270 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8301ceb7-def3-4e75-b283-ebdeaca9a877" containerName="main" Jul 22 05:16:48.068307 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068293 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8301ceb7-def3-4e75-b283-ebdeaca9a877" containerName="main" Jul 22 05:16:48.068488 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068325 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff8d3b73-9ee3-40e2-8fdf-4277435fd63e" containerName="cleanup" Jul 22 05:16:48.068488 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068333 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff8d3b73-9ee3-40e2-8fdf-4277435fd63e" containerName="cleanup" Jul 22 05:16:48.068488 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068349 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c84346-729d-4da9-83f1-174ad9d09561" containerName="cleanup" Jul 22 05:16:48.068488 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068359 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c84346-729d-4da9-83f1-174ad9d09561" containerName="cleanup" Jul 22 05:16:48.068488 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068392 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e8f4464-96a5-4e1f-801b-345b81804c4b" containerName="main" Jul 22 05:16:48.068488 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068401 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e8f4464-96a5-4e1f-801b-345b81804c4b" containerName="main" Jul 22 05:16:48.068789 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068499 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8301ceb7-def3-4e75-b283-ebdeaca9a877" containerName="main" Jul 22 05:16:48.068789 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068527 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e8f4464-96a5-4e1f-801b-345b81804c4b" containerName="main" Jul 22 05:16:48.068789 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068536 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="75c84346-729d-4da9-83f1-174ad9d09561" containerName="cleanup" Jul 22 05:16:48.068789 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.068545 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff8d3b73-9ee3-40e2-8fdf-4277435fd63e" containerName="cleanup" Jul 22 05:16:48.071719 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.071698 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.075075 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.075055 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1a3bbb\"/\"default-dockercfg-dmxqm\"" Jul 22 05:16:48.075200 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.075057 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1a3bbb\"/\"sub-test-model-1a3bbb-kserve-self-signed-certs\"" Jul 22 05:16:48.075320 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.075305 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1a3bbb\"/\"kube-root-ca.crt\"" Jul 22 05:16:48.075365 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.075319 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1a3bbb\"/\"openshift-service-ca.crt\"" Jul 22 05:16:48.079474 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.079454 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk"] Jul 22 05:16:48.095531 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.095509 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-model-cache\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.095625 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.095558 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c6121-a051-4fed-8a30-9cdcd2f6295f-tls-certs\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.095625 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.095605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-dshm\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.095721 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.095640 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-tmp-dir\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.095721 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.095684 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrzt\" (UniqueName: \"kubernetes.io/projected/910c6121-a051-4fed-8a30-9cdcd2f6295f-kube-api-access-4jrzt\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.095721 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.095704 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-home\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196408 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196362 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-tmp-dir\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196559 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196421 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrzt\" (UniqueName: \"kubernetes.io/projected/910c6121-a051-4fed-8a30-9cdcd2f6295f-kube-api-access-4jrzt\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196559 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-home\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196559 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196480 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-model-cache\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196559 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c6121-a051-4fed-8a30-9cdcd2f6295f-tls-certs\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196559 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196551 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-dshm\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196814 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196757 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-tmp-dir\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196884 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196811 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-model-cache\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.196940 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.196912 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-home\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.198725 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.198699 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-dshm\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.199029 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.199005 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c6121-a051-4fed-8a30-9cdcd2f6295f-tls-certs\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.203900 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.203883 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrzt\" (UniqueName: \"kubernetes.io/projected/910c6121-a051-4fed-8a30-9cdcd2f6295f-kube-api-access-4jrzt\") pod \"sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.382235 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.382161 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:48.512824 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.512788 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk"] Jul 22 05:16:48.513558 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:16:48.513537 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod910c6121_a051_4fed_8a30_9cdcd2f6295f.slice/crio-cd269972e91897a3db3cd8d4c7f401d573b4e50cb600c093df8e7f756bebd326 WatchSource:0}: Error finding container cd269972e91897a3db3cd8d4c7f401d573b4e50cb600c093df8e7f756bebd326: Status 404 returned error can't find the container with id cd269972e91897a3db3cd8d4c7f401d573b4e50cb600c093df8e7f756bebd326 Jul 22 05:16:48.749020 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.748934 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" event={"ID":"910c6121-a051-4fed-8a30-9cdcd2f6295f","Type":"ContainerStarted","Data":"2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e"} Jul 22 05:16:48.749020 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.748975 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" event={"ID":"910c6121-a051-4fed-8a30-9cdcd2f6295f","Type":"ContainerStarted","Data":"cd269972e91897a3db3cd8d4c7f401d573b4e50cb600c093df8e7f756bebd326"} Jul 22 05:16:48.765592 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:48.765545 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" podStartSLOduration=0.765532372 podStartE2EDuration="765.532372ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:16:48.763979027 +0000 UTC m=+2620.482584163" watchObservedRunningTime="2026-07-22 05:16:48.765532372 +0000 UTC m=+2620.484137506" Jul 22 05:16:51.659460 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:51.659421 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:16:58.382403 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:58.382355 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:58.382815 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:58.382416 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:58.395440 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:58.395416 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:16:58.814834 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:16:58.814803 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:17:07.557165 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:07.557131 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:17:09.098352 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:09.098321 2572 scope.go:117] "RemoveContainer" containerID="2f2c60c02b4c5593487616f5de9e30f681e86182d3de5a3abb76139981cf0f90" Jul 22 05:17:09.106635 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:09.106614 2572 scope.go:117] "RemoveContainer" containerID="f8409242a2e83343344ee618670a7e31daed900166bd53cab735a67502238029" Jul 22 05:17:26.366075 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:26.366039 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk"] Jul 22 05:17:32.560636 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:32.560594 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:17:35.844831 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.844794 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885"] Jul 22 05:17:35.848262 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.848246 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:35.851393 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.851360 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c0234d\"/\"default-dockercfg-tg6lh\"" Jul 22 05:17:35.851527 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.851441 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c0234d\"/\"openshift-service-ca.crt\"" Jul 22 05:17:35.851527 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.851363 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c0234d\"/\"rate-test-model-c0234d-kserve-self-signed-certs\"" Jul 22 05:17:35.851527 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.851363 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c0234d\"/\"kube-root-ca.crt\"" Jul 22 05:17:35.855664 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.855346 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885"] Jul 22 05:17:35.992686 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.992646 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-tmp-dir\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:35.992686 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.992682 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-dshm\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:35.992871 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.992706 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96941308-22c6-4ebf-9244-976fae014c63-tls-certs\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:35.992871 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.992750 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjf2s\" (UniqueName: \"kubernetes.io/projected/96941308-22c6-4ebf-9244-976fae014c63-kube-api-access-cjf2s\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:35.992871 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.992780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-model-cache\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:35.992871 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:35.992816 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-home\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.093538 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.093502 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-home\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.093702 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.093596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-tmp-dir\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.093702 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.093621 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-dshm\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.093702 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.093646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96941308-22c6-4ebf-9244-976fae014c63-tls-certs\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.093702 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.093677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjf2s\" (UniqueName: \"kubernetes.io/projected/96941308-22c6-4ebf-9244-976fae014c63-kube-api-access-cjf2s\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.093921 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.093715 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-model-cache\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.093921 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.093881 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-home\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.094036 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.093942 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-tmp-dir\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.094098 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.094061 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-model-cache\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.095860 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.095807 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-dshm\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.096100 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.096083 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96941308-22c6-4ebf-9244-976fae014c63-tls-certs\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.101249 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.101232 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjf2s\" (UniqueName: \"kubernetes.io/projected/96941308-22c6-4ebf-9244-976fae014c63-kube-api-access-cjf2s\") pod \"rate-test-model-c0234d-kserve-5d57d9dd7f-mj885\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.158054 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.158024 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:36.291055 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:36.291019 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885"] Jul 22 05:17:36.291905 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:17:36.291878 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96941308_22c6_4ebf_9244_976fae014c63.slice/crio-178c803b219571ed91e358d97b4bd7a2c9d8198f408af1473f6f96c3268e6fef WatchSource:0}: Error finding container 178c803b219571ed91e358d97b4bd7a2c9d8198f408af1473f6f96c3268e6fef: Status 404 returned error can't find the container with id 178c803b219571ed91e358d97b4bd7a2c9d8198f408af1473f6f96c3268e6fef Jul 22 05:17:37.003856 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:37.003818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" event={"ID":"96941308-22c6-4ebf-9244-976fae014c63","Type":"ContainerStarted","Data":"a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd"} Jul 22 05:17:37.003856 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:37.003869 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" event={"ID":"96941308-22c6-4ebf-9244-976fae014c63","Type":"ContainerStarted","Data":"178c803b219571ed91e358d97b4bd7a2c9d8198f408af1473f6f96c3268e6fef"} Jul 22 05:17:37.021877 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:37.021823 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" podStartSLOduration=2.021808371 podStartE2EDuration="2.021808371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:17:37.018503032 +0000 UTC m=+2668.737108166" watchObservedRunningTime="2026-07-22 05:17:37.021808371 +0000 UTC m=+2668.740413506" Jul 22 05:17:41.386272 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.386230 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" podUID="910c6121-a051-4fed-8a30-9cdcd2f6295f" containerName="main" containerID="cri-o://2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e" gracePeriod=45 Jul 22 05:17:41.628721 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.628693 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:17:41.741135 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741049 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-model-cache\") pod \"910c6121-a051-4fed-8a30-9cdcd2f6295f\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " Jul 22 05:17:41.741135 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741113 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c6121-a051-4fed-8a30-9cdcd2f6295f-tls-certs\") pod \"910c6121-a051-4fed-8a30-9cdcd2f6295f\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " Jul 22 05:17:41.741135 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741136 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-tmp-dir\") pod \"910c6121-a051-4fed-8a30-9cdcd2f6295f\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " Jul 22 05:17:41.741447 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741154 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-home\") pod \"910c6121-a051-4fed-8a30-9cdcd2f6295f\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " Jul 22 05:17:41.741447 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741192 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jrzt\" (UniqueName: \"kubernetes.io/projected/910c6121-a051-4fed-8a30-9cdcd2f6295f-kube-api-access-4jrzt\") pod \"910c6121-a051-4fed-8a30-9cdcd2f6295f\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " Jul 22 05:17:41.741447 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741228 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-dshm\") pod \"910c6121-a051-4fed-8a30-9cdcd2f6295f\" (UID: \"910c6121-a051-4fed-8a30-9cdcd2f6295f\") " Jul 22 05:17:41.741447 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741398 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-model-cache" (OuterVolumeSpecName: "model-cache") pod "910c6121-a051-4fed-8a30-9cdcd2f6295f" (UID: "910c6121-a051-4fed-8a30-9cdcd2f6295f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:17:41.741447 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741424 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "910c6121-a051-4fed-8a30-9cdcd2f6295f" (UID: "910c6121-a051-4fed-8a30-9cdcd2f6295f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:17:41.741625 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741480 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-home" (OuterVolumeSpecName: "home") pod "910c6121-a051-4fed-8a30-9cdcd2f6295f" (UID: "910c6121-a051-4fed-8a30-9cdcd2f6295f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:17:41.741625 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741596 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-model-cache\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:17:41.741692 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741640 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-tmp-dir\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:17:41.741692 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.741653 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-home\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:17:41.743335 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.743310 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-dshm" (OuterVolumeSpecName: "dshm") pod "910c6121-a051-4fed-8a30-9cdcd2f6295f" (UID: "910c6121-a051-4fed-8a30-9cdcd2f6295f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:17:41.743422 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.743349 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/910c6121-a051-4fed-8a30-9cdcd2f6295f-kube-api-access-4jrzt" (OuterVolumeSpecName: "kube-api-access-4jrzt") pod "910c6121-a051-4fed-8a30-9cdcd2f6295f" (UID: "910c6121-a051-4fed-8a30-9cdcd2f6295f"). InnerVolumeSpecName "kube-api-access-4jrzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:17:41.743460 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.743432 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/910c6121-a051-4fed-8a30-9cdcd2f6295f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "910c6121-a051-4fed-8a30-9cdcd2f6295f" (UID: "910c6121-a051-4fed-8a30-9cdcd2f6295f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 05:17:41.842535 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.842499 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/910c6121-a051-4fed-8a30-9cdcd2f6295f-dshm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:17:41.842535 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.842530 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/910c6121-a051-4fed-8a30-9cdcd2f6295f-tls-certs\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:17:41.842535 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:41.842542 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrzt\" (UniqueName: \"kubernetes.io/projected/910c6121-a051-4fed-8a30-9cdcd2f6295f-kube-api-access-4jrzt\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:17:42.037004 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.036970 2572 generic.go:356] "Generic (PLEG): container finished" podID="910c6121-a051-4fed-8a30-9cdcd2f6295f" containerID="2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e" exitCode=0 Jul 22 05:17:42.037170 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.037018 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" Jul 22 05:17:42.037170 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.037059 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" event={"ID":"910c6121-a051-4fed-8a30-9cdcd2f6295f","Type":"ContainerDied","Data":"2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e"} Jul 22 05:17:42.037170 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.037109 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk" event={"ID":"910c6121-a051-4fed-8a30-9cdcd2f6295f","Type":"ContainerDied","Data":"cd269972e91897a3db3cd8d4c7f401d573b4e50cb600c093df8e7f756bebd326"} Jul 22 05:17:42.037170 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.037133 2572 scope.go:117] "RemoveContainer" containerID="2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e" Jul 22 05:17:42.047250 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.047226 2572 scope.go:117] "RemoveContainer" containerID="2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e" Jul 22 05:17:42.047544 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:17:42.047525 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e\": container with ID starting with 2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e not found: ID does not exist" containerID="2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e" Jul 22 05:17:42.047622 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.047552 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e"} err="failed to get container status \"2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e\": rpc error: code = NotFound desc = could not find container \"2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e\": container with ID starting with 2379084c16b383e147a5ef1d69a1b90239227d8063ab5dca892f1955d85a0c5e not found: ID does not exist" Jul 22 05:17:42.058240 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.058217 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk"] Jul 22 05:17:42.062584 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.062564 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1a3bbb/sub-test-model-1a3bbb-kserve-f54d89ddf-sjttk"] Jul 22 05:17:42.801447 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:42.801415 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="910c6121-a051-4fed-8a30-9cdcd2f6295f" path="/var/lib/kubelet/pods/910c6121-a051-4fed-8a30-9cdcd2f6295f/volumes" Jul 22 05:17:46.158760 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:46.158727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:46.158760 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:46.158765 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:46.171351 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:46.171327 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:47.075447 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:47.075413 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:17:48.062624 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:17:48.062593 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:18:09.002699 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:09.002616 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:18:09.009937 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:09.009906 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:18:09.137672 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:09.137638 2572 scope.go:117] "RemoveContainer" containerID="e1a84d29da593e1dc9211a954bd07007b19ac859a8c70e166646bf0b0a9d7b13" Jul 22 05:18:14.456569 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:14.456535 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:18:29.565509 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:29.565472 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:18:47.064801 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:47.064765 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:18:53.145773 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:53.145732 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885"] Jul 22 05:18:55.193292 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.193257 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-139631815388848"] Jul 22 05:18:55.193671 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.193633 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="910c6121-a051-4fed-8a30-9cdcd2f6295f" containerName="main" Jul 22 05:18:55.193671 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.193647 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="910c6121-a051-4fed-8a30-9cdcd2f6295f" containerName="main" Jul 22 05:18:55.193746 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.193724 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="910c6121-a051-4fed-8a30-9cdcd2f6295f" containerName="main" Jul 22 05:18:55.196770 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.196753 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:55.199550 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.199523 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pz2cr\"" Jul 22 05:18:55.201914 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.201888 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-139631815388848"] Jul 22 05:18:55.254964 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.254927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwzw4\" (UniqueName: \"kubernetes.io/projected/9e21a3f1-5891-4c15-b955-cf7400c4c7c7-kube-api-access-lwzw4\") pod \"test-curl-7614-139631815388848\" (UID: \"9e21a3f1-5891-4c15-b955-cf7400c4c7c7\") " pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:55.355534 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.355497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwzw4\" (UniqueName: \"kubernetes.io/projected/9e21a3f1-5891-4c15-b955-cf7400c4c7c7-kube-api-access-lwzw4\") pod \"test-curl-7614-139631815388848\" (UID: \"9e21a3f1-5891-4c15-b955-cf7400c4c7c7\") " pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:55.363987 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.363963 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwzw4\" (UniqueName: \"kubernetes.io/projected/9e21a3f1-5891-4c15-b955-cf7400c4c7c7-kube-api-access-lwzw4\") pod \"test-curl-7614-139631815388848\" (UID: \"9e21a3f1-5891-4c15-b955-cf7400c4c7c7\") " pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:55.506621 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.506562 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:55.633893 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.633861 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-139631815388848"] Jul 22 05:18:55.635363 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:18:55.635338 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e21a3f1_5891_4c15_b955_cf7400c4c7c7.slice/crio-5dedab253df0de95911c724d453f2ce0bbd1cc2a45ab3cc6dfd5e61dad9b521c WatchSource:0}: Error finding container 5dedab253df0de95911c724d453f2ce0bbd1cc2a45ab3cc6dfd5e61dad9b521c: Status 404 returned error can't find the container with id 5dedab253df0de95911c724d453f2ce0bbd1cc2a45ab3cc6dfd5e61dad9b521c Jul 22 05:18:55.636775 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:55.636758 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 05:18:56.440719 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:56.440684 2572 generic.go:356] "Generic (PLEG): container finished" podID="9e21a3f1-5891-4c15-b955-cf7400c4c7c7" containerID="cb974e904f7d446d139c6e387ff8185fb2cf867b10eaee786a879ef324b73c05" exitCode=0 Jul 22 05:18:56.441044 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:56.440760 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-139631815388848" event={"ID":"9e21a3f1-5891-4c15-b955-cf7400c4c7c7","Type":"ContainerDied","Data":"cb974e904f7d446d139c6e387ff8185fb2cf867b10eaee786a879ef324b73c05"} Jul 22 05:18:56.441044 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:56.440802 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-139631815388848" event={"ID":"9e21a3f1-5891-4c15-b955-cf7400c4c7c7","Type":"ContainerStarted","Data":"5dedab253df0de95911c724d453f2ce0bbd1cc2a45ab3cc6dfd5e61dad9b521c"} Jul 22 05:18:57.575911 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:57.575883 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:57.609047 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:57.609021 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-139631815388848_9e21a3f1-5891-4c15-b955-cf7400c4c7c7/test-curl-7614-139631815388848/0.log" Jul 22 05:18:57.630980 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:57.630954 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-139631815388848"] Jul 22 05:18:57.633031 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:57.633011 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-139631815388848"] Jul 22 05:18:57.673370 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:57.673342 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwzw4\" (UniqueName: \"kubernetes.io/projected/9e21a3f1-5891-4c15-b955-cf7400c4c7c7-kube-api-access-lwzw4\") pod \"9e21a3f1-5891-4c15-b955-cf7400c4c7c7\" (UID: \"9e21a3f1-5891-4c15-b955-cf7400c4c7c7\") " Jul 22 05:18:57.675439 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:57.675419 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e21a3f1-5891-4c15-b955-cf7400c4c7c7-kube-api-access-lwzw4" (OuterVolumeSpecName: "kube-api-access-lwzw4") pod "9e21a3f1-5891-4c15-b955-cf7400c4c7c7" (UID: "9e21a3f1-5891-4c15-b955-cf7400c4c7c7"). InnerVolumeSpecName "kube-api-access-lwzw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:18:57.774326 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:57.774300 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwzw4\" (UniqueName: \"kubernetes.io/projected/9e21a3f1-5891-4c15-b955-cf7400c4c7c7-kube-api-access-lwzw4\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:18:58.452458 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:58.452428 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dedab253df0de95911c724d453f2ce0bbd1cc2a45ab3cc6dfd5e61dad9b521c" Jul 22 05:18:58.452458 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:58.452445 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:58.465391 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:18:58.465347 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-139631815388848\" is forbidden: User \"system:node:ip-10-0-142-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-47.ec2.internal' and this object" podUID="9e21a3f1-5891-4c15-b955-cf7400c4c7c7" pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:58.799949 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:18:58.799911 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-139631815388848\" is forbidden: User \"system:node:ip-10-0-142-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-47.ec2.internal' and this object" podUID="9e21a3f1-5891-4c15-b955-cf7400c4c7c7" pod="opendatahub/test-curl-7614-139631815388848" Jul 22 05:18:58.800622 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:18:58.800601 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e21a3f1-5891-4c15-b955-cf7400c4c7c7" path="/var/lib/kubelet/pods/9e21a3f1-5891-4c15-b955-cf7400c4c7c7/volumes" Jul 22 05:19:01.253855 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.253817 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-139631815377952"] Jul 22 05:19:01.254227 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.254188 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e21a3f1-5891-4c15-b955-cf7400c4c7c7" containerName="test-curl-7614-139631815388848" Jul 22 05:19:01.254227 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.254200 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e21a3f1-5891-4c15-b955-cf7400c4c7c7" containerName="test-curl-7614-139631815388848" Jul 22 05:19:01.254305 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.254259 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e21a3f1-5891-4c15-b955-cf7400c4c7c7" containerName="test-curl-7614-139631815388848" Jul 22 05:19:01.257382 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.257355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815377952" Jul 22 05:19:01.259814 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.259790 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pz2cr\"" Jul 22 05:19:01.261592 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.261568 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-139631815377952"] Jul 22 05:19:01.304163 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.304133 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpkkn\" (UniqueName: \"kubernetes.io/projected/a2d7009e-1024-4c34-85aa-dfe79afae702-kube-api-access-rpkkn\") pod \"test-curl-7614-139631815377952\" (UID: \"a2d7009e-1024-4c34-85aa-dfe79afae702\") " pod="opendatahub/test-curl-7614-139631815377952" Jul 22 05:19:01.405602 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.405564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpkkn\" (UniqueName: \"kubernetes.io/projected/a2d7009e-1024-4c34-85aa-dfe79afae702-kube-api-access-rpkkn\") pod \"test-curl-7614-139631815377952\" (UID: \"a2d7009e-1024-4c34-85aa-dfe79afae702\") " pod="opendatahub/test-curl-7614-139631815377952" Jul 22 05:19:01.414222 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.414193 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpkkn\" (UniqueName: \"kubernetes.io/projected/a2d7009e-1024-4c34-85aa-dfe79afae702-kube-api-access-rpkkn\") pod \"test-curl-7614-139631815377952\" (UID: \"a2d7009e-1024-4c34-85aa-dfe79afae702\") " pod="opendatahub/test-curl-7614-139631815377952" Jul 22 05:19:01.569887 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.569856 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815377952" Jul 22 05:19:01.696806 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:01.696777 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-139631815377952"] Jul 22 05:19:01.697758 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:19:01.697718 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d7009e_1024_4c34_85aa_dfe79afae702.slice/crio-c2198b892dc58e6419236156c9fd80d59cca8fe54564a2ba8ec42bcd66517136 WatchSource:0}: Error finding container c2198b892dc58e6419236156c9fd80d59cca8fe54564a2ba8ec42bcd66517136: Status 404 returned error can't find the container with id c2198b892dc58e6419236156c9fd80d59cca8fe54564a2ba8ec42bcd66517136 Jul 22 05:19:02.475232 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:02.475197 2572 generic.go:356] "Generic (PLEG): container finished" podID="a2d7009e-1024-4c34-85aa-dfe79afae702" containerID="a04dfe1a2ea6170aceb648e84105b2ce381d5bc4d8ec21286539b76e55f7a3ae" exitCode=0 Jul 22 05:19:02.475627 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:02.475282 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-139631815377952" event={"ID":"a2d7009e-1024-4c34-85aa-dfe79afae702","Type":"ContainerDied","Data":"a04dfe1a2ea6170aceb648e84105b2ce381d5bc4d8ec21286539b76e55f7a3ae"} Jul 22 05:19:02.475627 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:02.475320 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-139631815377952" event={"ID":"a2d7009e-1024-4c34-85aa-dfe79afae702","Type":"ContainerStarted","Data":"c2198b892dc58e6419236156c9fd80d59cca8fe54564a2ba8ec42bcd66517136"} Jul 22 05:19:03.608720 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:03.608692 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815377952" Jul 22 05:19:03.626536 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:03.626504 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-139631815377952_a2d7009e-1024-4c34-85aa-dfe79afae702/test-curl-7614-139631815377952/0.log" Jul 22 05:19:03.649454 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:03.649424 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-139631815377952"] Jul 22 05:19:03.651598 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:03.651574 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-139631815377952"] Jul 22 05:19:03.722472 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:03.722447 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpkkn\" (UniqueName: \"kubernetes.io/projected/a2d7009e-1024-4c34-85aa-dfe79afae702-kube-api-access-rpkkn\") pod \"a2d7009e-1024-4c34-85aa-dfe79afae702\" (UID: \"a2d7009e-1024-4c34-85aa-dfe79afae702\") " Jul 22 05:19:03.724390 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:03.724356 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d7009e-1024-4c34-85aa-dfe79afae702-kube-api-access-rpkkn" (OuterVolumeSpecName: "kube-api-access-rpkkn") pod "a2d7009e-1024-4c34-85aa-dfe79afae702" (UID: "a2d7009e-1024-4c34-85aa-dfe79afae702"). InnerVolumeSpecName "kube-api-access-rpkkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:19:03.823543 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:03.823518 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpkkn\" (UniqueName: \"kubernetes.io/projected/a2d7009e-1024-4c34-85aa-dfe79afae702-kube-api-access-rpkkn\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:04.487172 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:04.487142 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815377952" Jul 22 05:19:04.487336 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:04.487175 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2198b892dc58e6419236156c9fd80d59cca8fe54564a2ba8ec42bcd66517136" Jul 22 05:19:04.496659 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:19:04.496619 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-139631815377952\" is forbidden: User \"system:node:ip-10-0-142-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-47.ec2.internal' and this object" podUID="a2d7009e-1024-4c34-85aa-dfe79afae702" pod="opendatahub/test-curl-7614-139631815377952" Jul 22 05:19:04.801496 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:04.801463 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d7009e-1024-4c34-85aa-dfe79afae702" path="/var/lib/kubelet/pods/a2d7009e-1024-4c34-85aa-dfe79afae702/volumes" Jul 22 05:19:05.963201 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:05.963153 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:19:06.217221 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.217131 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:06.217594 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.217575 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d7009e-1024-4c34-85aa-dfe79afae702" containerName="test-curl-7614-139631815377952" Jul 22 05:19:06.217700 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.217596 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d7009e-1024-4c34-85aa-dfe79afae702" containerName="test-curl-7614-139631815377952" Jul 22 05:19:06.217754 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.217718 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d7009e-1024-4c34-85aa-dfe79afae702" containerName="test-curl-7614-139631815377952" Jul 22 05:19:06.220842 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.220821 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:06.223235 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.223213 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pz2cr\"" Jul 22 05:19:06.226782 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.226755 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:06.341666 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.341641 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69vnw\" (UniqueName: \"kubernetes.io/projected/2ab516e4-c30c-4a6c-9867-698147757d49-kube-api-access-69vnw\") pod \"test-curl-7614-139631815378960\" (UID: \"2ab516e4-c30c-4a6c-9867-698147757d49\") " pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:06.442248 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.442214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69vnw\" (UniqueName: \"kubernetes.io/projected/2ab516e4-c30c-4a6c-9867-698147757d49-kube-api-access-69vnw\") pod \"test-curl-7614-139631815378960\" (UID: \"2ab516e4-c30c-4a6c-9867-698147757d49\") " pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:06.449896 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.449867 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69vnw\" (UniqueName: \"kubernetes.io/projected/2ab516e4-c30c-4a6c-9867-698147757d49-kube-api-access-69vnw\") pod \"test-curl-7614-139631815378960\" (UID: \"2ab516e4-c30c-4a6c-9867-698147757d49\") " pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:06.530806 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.530784 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:06.867736 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:06.867714 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:06.869466 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:19:06.869436 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ab516e4_c30c_4a6c_9867_698147757d49.slice/crio-90161f3d817f5e416247457c392c57c9771a34d936d6779b6f70ff11e7822d38 WatchSource:0}: Error finding container 90161f3d817f5e416247457c392c57c9771a34d936d6779b6f70ff11e7822d38: Status 404 returned error can't find the container with id 90161f3d817f5e416247457c392c57c9771a34d936d6779b6f70ff11e7822d38 Jul 22 05:19:07.504833 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:07.504800 2572 generic.go:356] "Generic (PLEG): container finished" podID="2ab516e4-c30c-4a6c-9867-698147757d49" containerID="ae2a18c06d8f23ec954405b38069e746cbaca39b9c6c3998791d09336d46bbc1" exitCode=0 Jul 22 05:19:07.505165 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:07.504888 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-139631815378960" event={"ID":"2ab516e4-c30c-4a6c-9867-698147757d49","Type":"ContainerDied","Data":"ae2a18c06d8f23ec954405b38069e746cbaca39b9c6c3998791d09336d46bbc1"} Jul 22 05:19:07.505165 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:07.504928 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-139631815378960" event={"ID":"2ab516e4-c30c-4a6c-9867-698147757d49","Type":"ContainerStarted","Data":"90161f3d817f5e416247457c392c57c9771a34d936d6779b6f70ff11e7822d38"} Jul 22 05:19:08.165564 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.165520 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" podUID="96941308-22c6-4ebf-9244-976fae014c63" containerName="main" containerID="cri-o://a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd" gracePeriod=45 Jul 22 05:19:08.417186 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.417129 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:19:08.473537 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473506 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-home\") pod \"96941308-22c6-4ebf-9244-976fae014c63\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " Jul 22 05:19:08.473668 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473564 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjf2s\" (UniqueName: \"kubernetes.io/projected/96941308-22c6-4ebf-9244-976fae014c63-kube-api-access-cjf2s\") pod \"96941308-22c6-4ebf-9244-976fae014c63\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " Jul 22 05:19:08.473668 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473596 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96941308-22c6-4ebf-9244-976fae014c63-tls-certs\") pod \"96941308-22c6-4ebf-9244-976fae014c63\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " Jul 22 05:19:08.473668 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473624 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-dshm\") pod \"96941308-22c6-4ebf-9244-976fae014c63\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " Jul 22 05:19:08.473668 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473647 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-model-cache\") pod \"96941308-22c6-4ebf-9244-976fae014c63\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " Jul 22 05:19:08.473887 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473691 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-tmp-dir\") pod \"96941308-22c6-4ebf-9244-976fae014c63\" (UID: \"96941308-22c6-4ebf-9244-976fae014c63\") " Jul 22 05:19:08.473887 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473760 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-home" (OuterVolumeSpecName: "home") pod "96941308-22c6-4ebf-9244-976fae014c63" (UID: "96941308-22c6-4ebf-9244-976fae014c63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:19:08.473887 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473880 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-home\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:08.474036 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473898 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-model-cache" (OuterVolumeSpecName: "model-cache") pod "96941308-22c6-4ebf-9244-976fae014c63" (UID: "96941308-22c6-4ebf-9244-976fae014c63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:19:08.474036 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.473967 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96941308-22c6-4ebf-9244-976fae014c63" (UID: "96941308-22c6-4ebf-9244-976fae014c63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:19:08.475749 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.475718 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96941308-22c6-4ebf-9244-976fae014c63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96941308-22c6-4ebf-9244-976fae014c63" (UID: "96941308-22c6-4ebf-9244-976fae014c63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 05:19:08.475847 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.475796 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96941308-22c6-4ebf-9244-976fae014c63-kube-api-access-cjf2s" (OuterVolumeSpecName: "kube-api-access-cjf2s") pod "96941308-22c6-4ebf-9244-976fae014c63" (UID: "96941308-22c6-4ebf-9244-976fae014c63"). InnerVolumeSpecName "kube-api-access-cjf2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:19:08.475847 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.475799 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-dshm" (OuterVolumeSpecName: "dshm") pod "96941308-22c6-4ebf-9244-976fae014c63" (UID: "96941308-22c6-4ebf-9244-976fae014c63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 05:19:08.512252 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.512224 2572 generic.go:356] "Generic (PLEG): container finished" podID="96941308-22c6-4ebf-9244-976fae014c63" containerID="a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd" exitCode=0 Jul 22 05:19:08.512608 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.512271 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" Jul 22 05:19:08.512608 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.512306 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" event={"ID":"96941308-22c6-4ebf-9244-976fae014c63","Type":"ContainerDied","Data":"a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd"} Jul 22 05:19:08.512608 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.512345 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885" event={"ID":"96941308-22c6-4ebf-9244-976fae014c63","Type":"ContainerDied","Data":"178c803b219571ed91e358d97b4bd7a2c9d8198f408af1473f6f96c3268e6fef"} Jul 22 05:19:08.512608 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.512362 2572 scope.go:117] "RemoveContainer" containerID="a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd" Jul 22 05:19:08.522524 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.522507 2572 scope.go:117] "RemoveContainer" containerID="a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd" Jul 22 05:19:08.522786 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:19:08.522763 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd\": container with ID starting with a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd not found: ID does not exist" containerID="a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd" Jul 22 05:19:08.522853 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.522793 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd"} err="failed to get container status \"a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd\": rpc error: code = NotFound desc = could not find container \"a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd\": container with ID starting with a24c0977e3e1687a75c770643cc567aa0665cde9af222d09ca752610eea8dccd not found: ID does not exist" Jul 22 05:19:08.536750 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.536723 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885"] Jul 22 05:19:08.541046 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.541018 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c0234d/rate-test-model-c0234d-kserve-5d57d9dd7f-mj885"] Jul 22 05:19:08.574844 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.574817 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjf2s\" (UniqueName: \"kubernetes.io/projected/96941308-22c6-4ebf-9244-976fae014c63-kube-api-access-cjf2s\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:08.574956 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.574847 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96941308-22c6-4ebf-9244-976fae014c63-tls-certs\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:08.574956 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.574864 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-dshm\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:08.574956 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.574875 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-model-cache\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:08.574956 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.574887 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96941308-22c6-4ebf-9244-976fae014c63-tmp-dir\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:08.650055 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.650033 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:08.668658 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.668597 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-139631815378960_2ab516e4-c30c-4a6c-9867-698147757d49/test-curl-7614-139631815378960/0.log" Jul 22 05:19:08.690184 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.690161 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:08.693308 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.693288 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:08.777331 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.777309 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69vnw\" (UniqueName: \"kubernetes.io/projected/2ab516e4-c30c-4a6c-9867-698147757d49-kube-api-access-69vnw\") pod \"2ab516e4-c30c-4a6c-9867-698147757d49\" (UID: \"2ab516e4-c30c-4a6c-9867-698147757d49\") " Jul 22 05:19:08.779355 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.779332 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab516e4-c30c-4a6c-9867-698147757d49-kube-api-access-69vnw" (OuterVolumeSpecName: "kube-api-access-69vnw") pod "2ab516e4-c30c-4a6c-9867-698147757d49" (UID: "2ab516e4-c30c-4a6c-9867-698147757d49"). InnerVolumeSpecName "kube-api-access-69vnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:19:08.802366 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.802345 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ab516e4-c30c-4a6c-9867-698147757d49" path="/var/lib/kubelet/pods/2ab516e4-c30c-4a6c-9867-698147757d49/volumes" Jul 22 05:19:08.802751 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.802738 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96941308-22c6-4ebf-9244-976fae014c63" path="/var/lib/kubelet/pods/96941308-22c6-4ebf-9244-976fae014c63/volumes" Jul 22 05:19:08.819409 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.819371 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:08.819702 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.819691 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96941308-22c6-4ebf-9244-976fae014c63" containerName="main" Jul 22 05:19:08.819749 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.819704 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="96941308-22c6-4ebf-9244-976fae014c63" containerName="main" Jul 22 05:19:08.819749 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.819733 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab516e4-c30c-4a6c-9867-698147757d49" containerName="test-curl-7614-139631815378960" Jul 22 05:19:08.819749 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.819739 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab516e4-c30c-4a6c-9867-698147757d49" containerName="test-curl-7614-139631815378960" Jul 22 05:19:08.819837 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.819804 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="96941308-22c6-4ebf-9244-976fae014c63" containerName="main" Jul 22 05:19:08.819837 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.819815 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab516e4-c30c-4a6c-9867-698147757d49" containerName="test-curl-7614-139631815378960" Jul 22 05:19:08.822992 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.822977 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:08.828457 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.828432 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:08.878657 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.878630 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfbd\" (UniqueName: \"kubernetes.io/projected/9ca8da2c-9777-4641-840c-53caae944f89-kube-api-access-dlfbd\") pod \"test-curl-7614-139631815378960\" (UID: \"9ca8da2c-9777-4641-840c-53caae944f89\") " pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:08.878768 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.878718 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69vnw\" (UniqueName: \"kubernetes.io/projected/2ab516e4-c30c-4a6c-9867-698147757d49-kube-api-access-69vnw\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:08.979890 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.979833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfbd\" (UniqueName: \"kubernetes.io/projected/9ca8da2c-9777-4641-840c-53caae944f89-kube-api-access-dlfbd\") pod \"test-curl-7614-139631815378960\" (UID: \"9ca8da2c-9777-4641-840c-53caae944f89\") " pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:08.987618 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:08.987594 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfbd\" (UniqueName: \"kubernetes.io/projected/9ca8da2c-9777-4641-840c-53caae944f89-kube-api-access-dlfbd\") pod \"test-curl-7614-139631815378960\" (UID: \"9ca8da2c-9777-4641-840c-53caae944f89\") " pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:09.133650 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:09.133626 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:09.162266 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:09.162245 2572 scope.go:117] "RemoveContainer" containerID="55e1efad024747ea89b1dd7985e6b58e15eb4866976672b866ba3fdb51744d47" Jul 22 05:19:09.261276 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:09.261251 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:09.262498 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:19:09.262476 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ca8da2c_9777_4641_840c_53caae944f89.slice/crio-9dc6ab48baed02d04b563a593031391d1985bc3c7d096415252193561568726d WatchSource:0}: Error finding container 9dc6ab48baed02d04b563a593031391d1985bc3c7d096415252193561568726d: Status 404 returned error can't find the container with id 9dc6ab48baed02d04b563a593031391d1985bc3c7d096415252193561568726d Jul 22 05:19:09.519192 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:09.519161 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:09.519192 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:09.519183 2572 scope.go:117] "RemoveContainer" containerID="ae2a18c06d8f23ec954405b38069e746cbaca39b9c6c3998791d09336d46bbc1" Jul 22 05:19:09.520638 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:09.520614 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-139631815378960" event={"ID":"9ca8da2c-9777-4641-840c-53caae944f89","Type":"ContainerStarted","Data":"9dc6ab48baed02d04b563a593031391d1985bc3c7d096415252193561568726d"} Jul 22 05:19:10.527432 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:10.527400 2572 generic.go:356] "Generic (PLEG): container finished" podID="9ca8da2c-9777-4641-840c-53caae944f89" containerID="c97b8e1f2af0b7ef3969890315f0459ba5c6d2a89669a5564646623ad16c3a7b" exitCode=0 Jul 22 05:19:10.527858 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:10.527536 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-139631815378960" event={"ID":"9ca8da2c-9777-4641-840c-53caae944f89","Type":"ContainerDied","Data":"c97b8e1f2af0b7ef3969890315f0459ba5c6d2a89669a5564646623ad16c3a7b"} Jul 22 05:19:11.695809 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.695784 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:11.716808 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.716766 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-139631815378960_9ca8da2c-9777-4641-840c-53caae944f89/test-curl-7614-139631815378960/0.log" Jul 22 05:19:11.742044 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.742014 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:11.745639 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.745614 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-139631815378960"] Jul 22 05:19:11.801266 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.801201 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlfbd\" (UniqueName: \"kubernetes.io/projected/9ca8da2c-9777-4641-840c-53caae944f89-kube-api-access-dlfbd\") pod \"9ca8da2c-9777-4641-840c-53caae944f89\" (UID: \"9ca8da2c-9777-4641-840c-53caae944f89\") " Jul 22 05:19:11.803158 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.803131 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca8da2c-9777-4641-840c-53caae944f89-kube-api-access-dlfbd" (OuterVolumeSpecName: "kube-api-access-dlfbd") pod "9ca8da2c-9777-4641-840c-53caae944f89" (UID: "9ca8da2c-9777-4641-840c-53caae944f89"). InnerVolumeSpecName "kube-api-access-dlfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 05:19:11.902317 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.902283 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfbd\" (UniqueName: \"kubernetes.io/projected/9ca8da2c-9777-4641-840c-53caae944f89-kube-api-access-dlfbd\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:11.938593 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.938562 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt"] Jul 22 05:19:11.938918 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.938906 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ca8da2c-9777-4641-840c-53caae944f89" containerName="test-curl-7614-139631815378960" Jul 22 05:19:11.938960 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.938920 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ca8da2c-9777-4641-840c-53caae944f89" containerName="test-curl-7614-139631815378960" Jul 22 05:19:11.939016 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.939006 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ca8da2c-9777-4641-840c-53caae944f89" containerName="test-curl-7614-139631815378960" Jul 22 05:19:11.942047 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.942032 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" Jul 22 05:19:11.943998 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.943974 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:19:11.944102 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.944003 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:19:11.948993 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:11.948971 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt"] Jul 22 05:19:12.003077 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.003048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c707158c-049f-4b1b-9bb5-98f033ba5699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt\" (UID: \"c707158c-049f-4b1b-9bb5-98f033ba5699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" Jul 22 05:19:12.104233 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.104153 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c707158c-049f-4b1b-9bb5-98f033ba5699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt\" (UID: \"c707158c-049f-4b1b-9bb5-98f033ba5699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" Jul 22 05:19:12.104813 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.104792 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c707158c-049f-4b1b-9bb5-98f033ba5699-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt\" (UID: \"c707158c-049f-4b1b-9bb5-98f033ba5699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" Jul 22 05:19:12.255164 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.255126 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" Jul 22 05:19:12.387554 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.387527 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt"] Jul 22 05:19:12.388909 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:19:12.388886 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc707158c_049f_4b1b_9bb5_98f033ba5699.slice/crio-a2f1f7fa829221b09849bea581365bde5a579fcd86fcc90c2b1166443b7d855f WatchSource:0}: Error finding container a2f1f7fa829221b09849bea581365bde5a579fcd86fcc90c2b1166443b7d855f: Status 404 returned error can't find the container with id a2f1f7fa829221b09849bea581365bde5a579fcd86fcc90c2b1166443b7d855f Jul 22 05:19:12.540702 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.540656 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" event={"ID":"c707158c-049f-4b1b-9bb5-98f033ba5699","Type":"ContainerStarted","Data":"a2f1f7fa829221b09849bea581365bde5a579fcd86fcc90c2b1166443b7d855f"} Jul 22 05:19:12.541928 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.541911 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dc6ab48baed02d04b563a593031391d1985bc3c7d096415252193561568726d" Jul 22 05:19:12.542011 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.541939 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:12.551753 ip-10-0-142-47 kubenswrapper[2572]: E0722 05:19:12.551717 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-139631815378960\" is forbidden: User \"system:node:ip-10-0-142-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-47.ec2.internal' and this object" podUID="9ca8da2c-9777-4641-840c-53caae944f89" pod="opendatahub/test-curl-7614-139631815378960" Jul 22 05:19:12.801560 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:12.801530 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ca8da2c-9777-4641-840c-53caae944f89" path="/var/lib/kubelet/pods/9ca8da2c-9777-4641-840c-53caae944f89/volumes" Jul 22 05:19:13.549565 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:13.549530 2572 generic.go:356] "Generic (PLEG): container finished" podID="c707158c-049f-4b1b-9bb5-98f033ba5699" containerID="5e9444fd5f84d2c3e874a7e0a7609010febe4c7038f280729319d51fee3a4f46" exitCode=0 Jul 22 05:19:13.549739 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:13.549623 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" event={"ID":"c707158c-049f-4b1b-9bb5-98f033ba5699","Type":"ContainerDied","Data":"5e9444fd5f84d2c3e874a7e0a7609010febe4c7038f280729319d51fee3a4f46"} Jul 22 05:19:14.690654 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:14.690630 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" Jul 22 05:19:14.830302 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:14.830227 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c707158c-049f-4b1b-9bb5-98f033ba5699-maas-api-service-ca\") pod \"c707158c-049f-4b1b-9bb5-98f033ba5699\" (UID: \"c707158c-049f-4b1b-9bb5-98f033ba5699\") " Jul 22 05:19:14.830611 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:14.830588 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c707158c-049f-4b1b-9bb5-98f033ba5699-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c707158c-049f-4b1b-9bb5-98f033ba5699" (UID: "c707158c-049f-4b1b-9bb5-98f033ba5699"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:19:14.931462 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:14.931434 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c707158c-049f-4b1b-9bb5-98f033ba5699-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:15.561280 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:15.561253 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" Jul 22 05:19:15.561461 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:15.561296 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt" event={"ID":"c707158c-049f-4b1b-9bb5-98f033ba5699","Type":"ContainerDied","Data":"a2f1f7fa829221b09849bea581365bde5a579fcd86fcc90c2b1166443b7d855f"} Jul 22 05:19:15.561461 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:15.561344 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2f1f7fa829221b09849bea581365bde5a579fcd86fcc90c2b1166443b7d855f" Jul 22 05:19:22.387115 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:22.387072 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7"] Jul 22 05:19:22.393669 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:22.393629 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-c0234d-29744955-f8sd7"] Jul 22 05:19:22.801000 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:22.800969 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75c84346-729d-4da9-83f1-174ad9d09561" path="/var/lib/kubelet/pods/75c84346-729d-4da9-83f1-174ad9d09561/volumes" Jul 22 05:19:26.055206 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:26.055173 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:19:27.021616 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:27.021583 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt"] Jul 22 05:19:27.023623 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:27.023602 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-c0234d-fsxjt"] Jul 22 05:19:28.801752 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:28.801708 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c707158c-049f-4b1b-9bb5-98f033ba5699" path="/var/lib/kubelet/pods/c707158c-049f-4b1b-9bb5-98f033ba5699/volumes" Jul 22 05:19:33.926088 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:33.925999 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc"] Jul 22 05:19:33.926551 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:33.926399 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c707158c-049f-4b1b-9bb5-98f033ba5699" containerName="revoke-keys" Jul 22 05:19:33.926551 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:33.926411 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c707158c-049f-4b1b-9bb5-98f033ba5699" containerName="revoke-keys" Jul 22 05:19:33.926551 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:33.926481 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c707158c-049f-4b1b-9bb5-98f033ba5699" containerName="revoke-keys" Jul 22 05:19:33.930927 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:33.930908 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" Jul 22 05:19:33.933653 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:33.933632 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4n2k7\"" Jul 22 05:19:33.934020 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:33.934000 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 05:19:33.936171 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:33.936143 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc"] Jul 22 05:19:34.101876 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:34.101837 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7fba1f-b840-4684-b202-58fdfc29ab3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc\" (UID: \"ad7fba1f-b840-4684-b202-58fdfc29ab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" Jul 22 05:19:34.202521 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:34.202434 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7fba1f-b840-4684-b202-58fdfc29ab3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc\" (UID: \"ad7fba1f-b840-4684-b202-58fdfc29ab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" Jul 22 05:19:34.203020 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:34.202998 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7fba1f-b840-4684-b202-58fdfc29ab3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc\" (UID: \"ad7fba1f-b840-4684-b202-58fdfc29ab3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" Jul 22 05:19:34.242601 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:34.242573 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" Jul 22 05:19:34.380442 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:34.380418 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc"] Jul 22 05:19:34.382042 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:19:34.382020 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7fba1f_b840_4684_b202_58fdfc29ab3c.slice/crio-f4f69e0ba6c991e58a567d5e1d5590bf7ab6a37ebf9a06a0c5cb7f79ecb4e7f9 WatchSource:0}: Error finding container f4f69e0ba6c991e58a567d5e1d5590bf7ab6a37ebf9a06a0c5cb7f79ecb4e7f9: Status 404 returned error can't find the container with id f4f69e0ba6c991e58a567d5e1d5590bf7ab6a37ebf9a06a0c5cb7f79ecb4e7f9 Jul 22 05:19:34.666169 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:34.666133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" event={"ID":"ad7fba1f-b840-4684-b202-58fdfc29ab3c","Type":"ContainerStarted","Data":"a0f7125d8638c9afeba2fa880453fa561b40e2e5e556ea862059fa85362a07a9"} Jul 22 05:19:34.666321 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:34.666191 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" event={"ID":"ad7fba1f-b840-4684-b202-58fdfc29ab3c","Type":"ContainerStarted","Data":"f4f69e0ba6c991e58a567d5e1d5590bf7ab6a37ebf9a06a0c5cb7f79ecb4e7f9"} Jul 22 05:19:34.681181 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:34.681126 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" podStartSLOduration=1.68111216 podStartE2EDuration="1.68111216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 05:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:19:34.679216173 +0000 UTC m=+2786.397821315" watchObservedRunningTime="2026-07-22 05:19:34.68111216 +0000 UTC m=+2786.399717294" Jul 22 05:19:35.672643 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:35.672604 2572 generic.go:356] "Generic (PLEG): container finished" podID="ad7fba1f-b840-4684-b202-58fdfc29ab3c" containerID="a0f7125d8638c9afeba2fa880453fa561b40e2e5e556ea862059fa85362a07a9" exitCode=0 Jul 22 05:19:35.673067 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:35.672668 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" event={"ID":"ad7fba1f-b840-4684-b202-58fdfc29ab3c","Type":"ContainerDied","Data":"a0f7125d8638c9afeba2fa880453fa561b40e2e5e556ea862059fa85362a07a9"} Jul 22 05:19:36.812324 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:36.812300 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" Jul 22 05:19:36.927692 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:36.927655 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7fba1f-b840-4684-b202-58fdfc29ab3c-maas-api-service-ca\") pod \"ad7fba1f-b840-4684-b202-58fdfc29ab3c\" (UID: \"ad7fba1f-b840-4684-b202-58fdfc29ab3c\") " Jul 22 05:19:36.928040 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:36.928015 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7fba1f-b840-4684-b202-58fdfc29ab3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad7fba1f-b840-4684-b202-58fdfc29ab3c" (UID: "ad7fba1f-b840-4684-b202-58fdfc29ab3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 05:19:37.029189 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:37.029159 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad7fba1f-b840-4684-b202-58fdfc29ab3c-maas-api-service-ca\") on node \"ip-10-0-142-47.ec2.internal\" DevicePath \"\"" Jul 22 05:19:37.684420 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:37.684366 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" event={"ID":"ad7fba1f-b840-4684-b202-58fdfc29ab3c","Type":"ContainerDied","Data":"f4f69e0ba6c991e58a567d5e1d5590bf7ab6a37ebf9a06a0c5cb7f79ecb4e7f9"} Jul 22 05:19:37.684420 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:37.684417 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc" Jul 22 05:19:37.684647 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:37.684433 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4f69e0ba6c991e58a567d5e1d5590bf7ab6a37ebf9a06a0c5cb7f79ecb4e7f9" Jul 22 05:19:44.368051 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:44.368020 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh"] Jul 22 05:19:44.371533 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:44.371508 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1a3bbb-29744955-9grjh"] Jul 22 05:19:44.801053 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:44.801024 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff8d3b73-9ee3-40e2-8fdf-4277435fd63e" path="/var/lib/kubelet/pods/ff8d3b73-9ee3-40e2-8fdf-4277435fd63e/volumes" Jul 22 05:19:47.660552 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:47.660503 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4msqs"] Jul 22 05:19:49.007427 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:49.007388 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc"] Jul 22 05:19:49.012701 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:49.012670 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1a3bbb-npqgc"] Jul 22 05:19:50.808825 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:19:50.808786 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7fba1f-b840-4684-b202-58fdfc29ab3c" path="/var/lib/kubelet/pods/ad7fba1f-b840-4684-b202-58fdfc29ab3c/volumes" Jul 22 05:20:02.309055 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:02.309024 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5567574457-b6l2k_0014fc51-7b57-47b0-9be1-62a7567fd2df/copy-manifests/0.log" Jul 22 05:20:02.315805 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:02.315768 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5567574457-b6l2k_0014fc51-7b57-47b0-9be1-62a7567fd2df/manager/0.log" Jul 22 05:20:02.895183 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:02.895151 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744925-tspcf_b844ba8d-2d5c-4680-a624-154807fe27bf/cleanup/0.log" Jul 22 05:20:03.004689 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:03.004657 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744940-x8spc_4254c65b-0c78-483d-af86-80415362df58/cleanup/0.log" Jul 22 05:20:04.040929 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.040898 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6_7d143919-7db5-45a8-91e9-04ae1d5ff7e7/util/0.log" Jul 22 05:20:04.046925 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.046904 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6_7d143919-7db5-45a8-91e9-04ae1d5ff7e7/pull/0.log" Jul 22 05:20:04.053218 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.053200 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6_7d143919-7db5-45a8-91e9-04ae1d5ff7e7/extract/0.log" Jul 22 05:20:04.176743 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.176719 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg_f07d86d0-7d97-4657-9d50-c7cd80fe16a9/util/0.log" Jul 22 05:20:04.182711 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.182695 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg_f07d86d0-7d97-4657-9d50-c7cd80fe16a9/pull/0.log" Jul 22 05:20:04.188496 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.188466 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg_f07d86d0-7d97-4657-9d50-c7cd80fe16a9/extract/0.log" Jul 22 05:20:04.309333 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.309270 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb_c49ffe21-6dc2-4c4a-997f-5151252bc373/util/0.log" Jul 22 05:20:04.315469 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.315440 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb_c49ffe21-6dc2-4c4a-997f-5151252bc373/pull/0.log" Jul 22 05:20:04.321933 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.321918 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb_c49ffe21-6dc2-4c4a-997f-5151252bc373/extract/0.log" Jul 22 05:20:04.434182 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.434154 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh_9d0bcbfb-e89d-4952-a244-3d684923ac79/extract/0.log" Jul 22 05:20:04.440175 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.440152 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh_9d0bcbfb-e89d-4952-a244-3d684923ac79/util/0.log" Jul 22 05:20:04.445718 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:04.445701 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh_9d0bcbfb-e89d-4952-a244-3d684923ac79/pull/0.log" Jul 22 05:20:05.315280 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:05.315256 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4msqs_2fe640d5-0ec1-475d-bb87-5d0deaa29ed9/limitador/0.log" Jul 22 05:20:05.959128 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:05.959101 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4gkls_04765fdd-a889-49af-b3eb-2d0fa42230cd/discovery/0.log" Jul 22 05:20:06.579385 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:06.579345 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vgrwr_1218d7b6-1165-4ae1-a285-a85ea651eb82/payload-processing/0.log" Jul 22 05:20:06.696852 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:06.696827 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-dcfd7dc78-8zf2f_e246d46d-6258-4941-a2c0-c71e777bf0ad/router/0.log" Jul 22 05:20:07.405928 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:07.405901 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgq2ss_0adfbec2-eb35-433c-836b-2e9ca3ed7752/main/0.log" Jul 22 05:20:16.534809 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:16.534776 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrkws_9c051ba5-d2d1-4add-89e0-591295ecaf5e/global-pull-secret-syncer/0.log" Jul 22 05:20:16.656970 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:16.656940 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t5hss_66acea06-2cd5-4939-bca5-b63eda7f5b29/konnectivity-agent/0.log" Jul 22 05:20:16.768197 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:16.768170 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-47.ec2.internal_480d41ccf0f1f19198559f5c186d8771/haproxy/0.log" Jul 22 05:20:21.479933 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.479902 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6_7d143919-7db5-45a8-91e9-04ae1d5ff7e7/extract/0.log" Jul 22 05:20:21.501418 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.501390 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6_7d143919-7db5-45a8-91e9-04ae1d5ff7e7/util/0.log" Jul 22 05:20:21.522886 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.522868 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599x8g6_7d143919-7db5-45a8-91e9-04ae1d5ff7e7/pull/0.log" Jul 22 05:20:21.548945 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.548922 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg_f07d86d0-7d97-4657-9d50-c7cd80fe16a9/extract/0.log" Jul 22 05:20:21.574260 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.574239 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg_f07d86d0-7d97-4657-9d50-c7cd80fe16a9/util/0.log" Jul 22 05:20:21.608817 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.608792 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05q6xg_f07d86d0-7d97-4657-9d50-c7cd80fe16a9/pull/0.log" Jul 22 05:20:21.638704 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.638686 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb_c49ffe21-6dc2-4c4a-997f-5151252bc373/extract/0.log" Jul 22 05:20:21.663058 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.663038 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb_c49ffe21-6dc2-4c4a-997f-5151252bc373/util/0.log" Jul 22 05:20:21.688545 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.688523 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnkxb_c49ffe21-6dc2-4c4a-997f-5151252bc373/pull/0.log" Jul 22 05:20:21.733896 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.733847 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh_9d0bcbfb-e89d-4952-a244-3d684923ac79/extract/0.log" Jul 22 05:20:21.757838 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.757814 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh_9d0bcbfb-e89d-4952-a244-3d684923ac79/util/0.log" Jul 22 05:20:21.778765 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:21.778748 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef147mrh_9d0bcbfb-e89d-4952-a244-3d684923ac79/pull/0.log" Jul 22 05:20:22.826054 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:22.826028 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4msqs_2fe640d5-0ec1-475d-bb87-5d0deaa29ed9/limitador/0.log" Jul 22 05:20:24.479293 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.479265 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8323e423-1641-434c-ae24-8161a3dbb46e/alertmanager/0.log" Jul 22 05:20:24.506690 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.506665 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8323e423-1641-434c-ae24-8161a3dbb46e/config-reloader/0.log" Jul 22 05:20:24.530662 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.530644 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8323e423-1641-434c-ae24-8161a3dbb46e/kube-rbac-proxy-web/0.log" Jul 22 05:20:24.555320 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.555297 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8323e423-1641-434c-ae24-8161a3dbb46e/kube-rbac-proxy/0.log" Jul 22 05:20:24.578549 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.578529 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8323e423-1641-434c-ae24-8161a3dbb46e/kube-rbac-proxy-metric/0.log" Jul 22 05:20:24.601625 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.601603 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8323e423-1641-434c-ae24-8161a3dbb46e/prom-label-proxy/0.log" Jul 22 05:20:24.629236 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.629214 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8323e423-1641-434c-ae24-8161a3dbb46e/init-config-reloader/0.log" Jul 22 05:20:24.670200 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.670178 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-8lqlm_a19fb360-7ca0-490a-a7a6-72497334a8ba/cluster-monitoring-operator/0.log" Jul 22 05:20:24.876837 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.876810 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-92jgj_e3847ee0-c179-4e80-a233-f9c66722e203/node-exporter/0.log" Jul 22 05:20:24.898817 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.898793 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-92jgj_e3847ee0-c179-4e80-a233-f9c66722e203/kube-rbac-proxy/0.log" Jul 22 05:20:24.930793 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:24.930770 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-92jgj_e3847ee0-c179-4e80-a233-f9c66722e203/init-textfile/0.log" Jul 22 05:20:25.131730 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:25.131661 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-h8tfm_ca60eca3-d74f-46bb-82eb-e6d4d7fd2781/kube-rbac-proxy-main/0.log" Jul 22 05:20:25.153392 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:25.153350 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-h8tfm_ca60eca3-d74f-46bb-82eb-e6d4d7fd2781/kube-rbac-proxy-self/0.log" Jul 22 05:20:25.179877 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:25.179841 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-h8tfm_ca60eca3-d74f-46bb-82eb-e6d4d7fd2781/openshift-state-metrics/0.log" Jul 22 05:20:25.379999 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:25.379972 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-nmxw4_fa088c82-ac38-444a-ac7e-23ac88cb6759/prometheus-operator/0.log" Jul 22 05:20:25.411172 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:25.411108 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-nmxw4_fa088c82-ac38-444a-ac7e-23ac88cb6759/kube-rbac-proxy/0.log" Jul 22 05:20:26.990784 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:26.990752 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-4qc7c_814f7eac-85c9-4308-a905-c09798acd37c/networking-console-plugin/0.log" Jul 22 05:20:27.634802 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.634772 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/2.log" Jul 22 05:20:27.638743 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.638721 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-fkwqf_a55ed3ab-0031-410b-bb8b-77c96710507b/console-operator/3.log" Jul 22 05:20:27.674737 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.674708 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t"] Jul 22 05:20:27.675058 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.675046 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7fba1f-b840-4684-b202-58fdfc29ab3c" containerName="revoke-keys" Jul 22 05:20:27.675101 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.675060 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7fba1f-b840-4684-b202-58fdfc29ab3c" containerName="revoke-keys" Jul 22 05:20:27.675140 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.675122 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7fba1f-b840-4684-b202-58fdfc29ab3c" containerName="revoke-keys" Jul 22 05:20:27.678190 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.678176 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.680224 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.680196 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-84vmg\"/\"openshift-service-ca.crt\"" Jul 22 05:20:27.680984 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.680963 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-84vmg\"/\"default-dockercfg-vt96m\"" Jul 22 05:20:27.681095 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.680993 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-84vmg\"/\"kube-root-ca.crt\"" Jul 22 05:20:27.685680 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.685634 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t"] Jul 22 05:20:27.846282 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.846243 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-sys\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.846497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.846299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdmwv\" (UniqueName: \"kubernetes.io/projected/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-kube-api-access-xdmwv\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.846497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.846358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-lib-modules\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.846497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.846448 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-podres\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.846497 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.846476 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-proc\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947220 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947131 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-podres\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947220 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947169 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-proc\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947220 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-sys\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947531 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdmwv\" (UniqueName: \"kubernetes.io/projected/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-kube-api-access-xdmwv\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947531 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947270 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-proc\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947531 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947296 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-podres\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947531 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947279 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-sys\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947531 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-lib-modules\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.947531 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.947492 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-lib-modules\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.955344 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.955319 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdmwv\" (UniqueName: \"kubernetes.io/projected/13b61fb5-ce2e-4729-9413-eac9a8f4f36e-kube-api-access-xdmwv\") pod \"perf-node-gather-daemonset-kpm7t\" (UID: \"13b61fb5-ce2e-4729-9413-eac9a8f4f36e\") " pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:27.989827 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:27.989804 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:28.121391 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:28.121348 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t"] Jul 22 05:20:28.123281 ip-10-0-142-47 kubenswrapper[2572]: W0722 05:20:28.123253 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod13b61fb5_ce2e_4729_9413_eac9a8f4f36e.slice/crio-cf21c2238e8573a61d23a8ce5553600108659d7cb5ccfc9ef78893a771785dab WatchSource:0}: Error finding container cf21c2238e8573a61d23a8ce5553600108659d7cb5ccfc9ef78893a771785dab: Status 404 returned error can't find the container with id cf21c2238e8573a61d23a8ce5553600108659d7cb5ccfc9ef78893a771785dab Jul 22 05:20:28.896998 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:28.896972 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-zhhft_18a09dee-af07-4b0a-9df9-ad8b3f6cd049/volume-data-source-validator/0.log" Jul 22 05:20:28.966012 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:28.965978 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" event={"ID":"13b61fb5-ce2e-4729-9413-eac9a8f4f36e","Type":"ContainerStarted","Data":"d907c8b78dd71c083900747ba0a66a7216ced75d7c91cfbcd971f080cd0098bf"} Jul 22 05:20:28.966206 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:28.966032 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" event={"ID":"13b61fb5-ce2e-4729-9413-eac9a8f4f36e","Type":"ContainerStarted","Data":"cf21c2238e8573a61d23a8ce5553600108659d7cb5ccfc9ef78893a771785dab"} Jul 22 05:20:28.966206 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:28.966101 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:28.981331 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:28.981282 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" podStartSLOduration=1.9812684950000001 podStartE2EDuration="1.981268495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 05:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 05:20:28.979257221 +0000 UTC m=+2840.697862383" watchObservedRunningTime="2026-07-22 05:20:28.981268495 +0000 UTC m=+2840.699873629" Jul 22 05:20:29.968870 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:29.968834 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h2wp5_edae6ef2-897e-4f71-8df9-66e34eaaff9c/dns/0.log" Jul 22 05:20:29.989908 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:29.989881 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h2wp5_edae6ef2-897e-4f71-8df9-66e34eaaff9c/kube-rbac-proxy/0.log" Jul 22 05:20:30.106748 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:30.106719 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k8rb5_227413da-239d-4d90-8c76-8861f42e0ac0/dns-node-resolver/0.log" Jul 22 05:20:30.703725 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:30.703697 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bmptl_d00b0010-35ef-4ad0-a483-937a188385cc/node-ca/0.log" Jul 22 05:20:31.823330 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:31.823297 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4gkls_04765fdd-a889-49af-b3eb-2d0fa42230cd/discovery/0.log" Jul 22 05:20:32.013755 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:32.013723 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vgrwr_1218d7b6-1165-4ae1-a285-a85ea651eb82/payload-processing/0.log" Jul 22 05:20:32.033320 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:32.033292 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-dcfd7dc78-8zf2f_e246d46d-6258-4941-a2c0-c71e777bf0ad/router/0.log" Jul 22 05:20:32.824339 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:32.824309 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x5p5s_48e78c79-6a2d-44cc-8f84-c123315037b7/serve-healthcheck-canary/0.log" Jul 22 05:20:33.289306 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:33.289279 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-gz2rz_02c59b58-bf2a-40bb-85dd-77a54ced505c/insights-operator/0.log" Jul 22 05:20:33.289986 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:33.289967 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-gz2rz_02c59b58-bf2a-40bb-85dd-77a54ced505c/insights-operator/1.log" Jul 22 05:20:33.456062 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:33.456033 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r26sh_d30365b3-2e9f-43bd-ac4c-afb30c8e228d/kube-rbac-proxy/0.log" Jul 22 05:20:33.477806 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:33.477782 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r26sh_d30365b3-2e9f-43bd-ac4c-afb30c8e228d/exporter/0.log" Jul 22 05:20:33.500448 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:33.500420 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r26sh_d30365b3-2e9f-43bd-ac4c-afb30c8e228d/extractor/0.log" Jul 22 05:20:34.981344 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:34.981319 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-84vmg/perf-node-gather-daemonset-kpm7t" Jul 22 05:20:36.515124 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:36.515096 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5567574457-b6l2k_0014fc51-7b57-47b0-9be1-62a7567fd2df/manager/0.log" Jul 22 05:20:36.534720 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:36.534692 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5567574457-b6l2k_0014fc51-7b57-47b0-9be1-62a7567fd2df/copy-manifests/0.log" Jul 22 05:20:38.091934 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:38.091905 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f987c9cb4-2fdzz_47b273c9-cbe8-4e85-9fd9-1d70e952a8ad/manager/0.log" Jul 22 05:20:38.843550 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:38.843519 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4gkls_04765fdd-a889-49af-b3eb-2d0fa42230cd/discovery/0.log" Jul 22 05:20:38.998321 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:38.998293 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vgrwr_1218d7b6-1165-4ae1-a285-a85ea651eb82/payload-processing/0.log" Jul 22 05:20:39.019608 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:39.019580 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-dcfd7dc78-8zf2f_e246d46d-6258-4941-a2c0-c71e777bf0ad/router/0.log" Jul 22 05:20:40.391593 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:40.391559 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-4gkls_04765fdd-a889-49af-b3eb-2d0fa42230cd/discovery/0.log" Jul 22 05:20:40.551951 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:40.551922 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-vgrwr_1218d7b6-1165-4ae1-a285-a85ea651eb82/payload-processing/0.log" Jul 22 05:20:40.567832 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:40.567810 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-dcfd7dc78-8zf2f_e246d46d-6258-4941-a2c0-c71e777bf0ad/router/0.log" Jul 22 05:20:43.650133 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:43.650006 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-mrht5_8e47c0c3-83ee-4f91-b30d-83203d3d2bb5/kube-storage-version-migrator-operator/1.log" Jul 22 05:20:43.650971 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:43.650952 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-mrht5_8e47c0c3-83ee-4f91-b30d-83203d3d2bb5/kube-storage-version-migrator-operator/0.log" Jul 22 05:20:44.660746 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:44.660716 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9nnqb_72bb9cb1-2e8a-44c3-8d61-043b7605a224/kube-multus-additional-cni-plugins/0.log" Jul 22 05:20:44.681622 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:44.681594 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9nnqb_72bb9cb1-2e8a-44c3-8d61-043b7605a224/egress-router-binary-copy/0.log" Jul 22 05:20:44.701452 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:44.701430 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9nnqb_72bb9cb1-2e8a-44c3-8d61-043b7605a224/cni-plugins/0.log" Jul 22 05:20:44.723516 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:44.723494 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9nnqb_72bb9cb1-2e8a-44c3-8d61-043b7605a224/bond-cni-plugin/0.log" Jul 22 05:20:44.743086 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:44.743065 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9nnqb_72bb9cb1-2e8a-44c3-8d61-043b7605a224/routeoverride-cni/0.log" Jul 22 05:20:44.764608 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:44.764585 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9nnqb_72bb9cb1-2e8a-44c3-8d61-043b7605a224/whereabouts-cni-bincopy/0.log" Jul 22 05:20:44.784298 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:44.784282 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9nnqb_72bb9cb1-2e8a-44c3-8d61-043b7605a224/whereabouts-cni/0.log" Jul 22 05:20:45.148728 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:45.148700 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zxf5g_63218f68-800c-4699-930f-b228bb95d1f4/kube-multus/0.log" Jul 22 05:20:45.254335 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:45.254307 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wsxd4_0c2de1e1-0496-4eda-b5e5-12b1c181582e/network-metrics-daemon/0.log" Jul 22 05:20:45.271757 ip-10-0-142-47 kubenswrapper[2572]: I0722 05:20:45.271735 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wsxd4_0c2de1e1-0496-4eda-b5e5-12b1c181582e/kube-rbac-proxy/0.log"