Jul 09 21:51:20.556865 ip-10-0-143-160 systemd[1]: Starting Kubernetes Kubelet... Jul 09 21:51:20.989978 ip-10-0-143-160 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 21:51:20.989978 ip-10-0-143-160 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 21:51:20.989978 ip-10-0-143-160 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 21:51:20.989978 ip-10-0-143-160 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 21:51:20.989978 ip-10-0-143-160 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 21:51:20.992887 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:20.992791 2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 21:51:20.998723 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998708 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 21:51:20.998723 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998723 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998727 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998731 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998735 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998738 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998742 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998746 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998749 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998752 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998755 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998759 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998762 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998765 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998768 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998771 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998774 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998777 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998781 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998784 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998787 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 21:51:20.998789 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998790 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998793 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998796 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998799 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998803 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998808 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998812 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998816 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998820 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998823 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998826 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998829 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998832 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998835 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998838 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998841 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998844 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998847 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998849 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 21:51:20.999327 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998852 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998855 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998858 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998860 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998863 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998866 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998868 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998871 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998874 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998878 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998883 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998887 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998891 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998895 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998899 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998905 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998910 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998914 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998917 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998920 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 21:51:20.999835 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998924 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998927 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998930 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998932 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998935 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998939 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998944 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998947 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998949 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998952 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998955 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998958 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998961 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998963 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998966 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998969 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998972 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998974 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998977 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 21:51:21.000356 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998980 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998983 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998986 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998989 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998991 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998994 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.998997 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999000 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999005 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999008 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999011 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999014 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999016 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999019 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999022 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999025 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999027 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999030 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999033 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999036 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 21:51:21.000878 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999839 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999847 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999852 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999856 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999859 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999863 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999866 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999869 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999872 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999875 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999878 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999881 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999884 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999887 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999891 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999894 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999897 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999899 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999902 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999905 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 21:51:21.001400 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999909 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999912 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999915 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999918 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999921 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999923 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999926 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999929 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999932 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999935 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999937 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999940 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999943 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999945 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999948 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999951 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999953 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999956 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999959 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 21:51:21.002279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999961 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999964 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999966 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999969 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999973 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999976 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999979 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999982 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999985 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999988 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999991 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999994 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999996 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:20.999999 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000002 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000004 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000007 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000010 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000013 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000016 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 21:51:21.003093 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000018 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000021 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000023 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000026 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000029 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000032 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000035 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000038 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000040 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000043 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000046 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000049 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000052 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000054 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000057 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000060 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000063 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000066 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000070 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000074 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 21:51:21.003623 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000077 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000080 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000083 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000086 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000089 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000092 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000094 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000097 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000100 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000103 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000106 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000108 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000111 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000115 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000118 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000123 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000126 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000129 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000132 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 21:51:21.004250 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000136 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000521 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000526 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000529 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000532 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000535 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000538 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000556 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000560 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000563 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000566 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000569 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000572 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000574 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000578 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000581 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000585 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000589 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000592 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 21:51:21.004973 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000596 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000599 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000602 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000604 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000607 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000610 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000613 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000615 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000618 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000621 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000623 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000626 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000629 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000632 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000635 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000638 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000640 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000643 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000663 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000667 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 21:51:21.005587 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000672 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000674 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000678 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000680 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000683 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000686 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000688 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000693 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000697 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000701 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000704 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000707 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000709 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000712 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000715 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000717 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000720 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000723 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000725 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 21:51:21.006138 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000728 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000730 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000733 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000736 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000739 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000741 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000744 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000747 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000750 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000753 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000755 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000758 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000761 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000765 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000768 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000771 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000774 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000777 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000779 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000782 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 21:51:21.006739 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000785 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000787 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000790 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000793 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000797 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000800 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000804 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000808 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000810 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000813 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000816 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000820 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000823 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000825 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000828 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000830 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000833 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000836 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000839 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000841 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 21:51:21.007255 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000844 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.000847 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000922 2556 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000932 2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000940 2556 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000949 2556 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000970 2556 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000974 2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000979 2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000983 2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000986 2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000989 2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000993 2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.000997 2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001001 2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001005 2556 flags.go:64] FLAG: --cgroup-root="" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001008 2556 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001011 2556 flags.go:64] FLAG: --client-ca-file="" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001014 2556 flags.go:64] FLAG: --cloud-provider="external" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001017 2556 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001023 2556 flags.go:64] FLAG: --cluster-domain="" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001026 2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001031 2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001034 2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 21:51:21.007813 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001038 2556 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001042 2556 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001045 2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001049 2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001052 2556 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001056 2556 flags.go:64] FLAG: --contention-profiling="false" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001059 2556 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001062 2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001066 2556 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001069 2556 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001074 2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001077 2556 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001080 2556 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001083 2556 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001087 2556 flags.go:64] FLAG: --enable-server="true" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001090 2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001095 2556 flags.go:64] FLAG: --event-burst="100" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001098 2556 flags.go:64] FLAG: --event-qps="50" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001101 2556 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001104 2556 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001107 2556 flags.go:64] FLAG: --eviction-hard="" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001111 2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001114 2556 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001117 2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001123 2556 flags.go:64] FLAG: --eviction-soft="" Jul 09 21:51:21.008409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001126 2556 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001130 2556 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001133 2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001136 2556 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001139 2556 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001144 2556 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001147 2556 flags.go:64] FLAG: --feature-gates="" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001151 2556 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001154 2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001158 2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001161 2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001164 2556 flags.go:64] FLAG: --healthz-port="10248" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001168 2556 flags.go:64] FLAG: --help="false" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001171 2556 flags.go:64] FLAG: --hostname-override="ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001174 2556 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001177 2556 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001180 2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001185 2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001189 2556 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001192 2556 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001195 2556 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001198 2556 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001201 2556 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 21:51:21.009103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001204 2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001208 2556 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001211 2556 flags.go:64] FLAG: --kube-reserved="" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001214 2556 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001217 2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001220 2556 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001223 2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001226 2556 flags.go:64] FLAG: --lock-file="" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001229 2556 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001233 2556 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001236 2556 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001242 2556 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001245 2556 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001248 2556 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001253 2556 flags.go:64] FLAG: --logging-format="text" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001256 2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001259 2556 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001262 2556 flags.go:64] FLAG: --manifest-url="" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001265 2556 flags.go:64] FLAG: --manifest-url-header="" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001270 2556 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001273 2556 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001277 2556 flags.go:64] FLAG: --max-pods="110" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001280 2556 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001284 2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001287 2556 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 21:51:21.009850 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001290 2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001293 2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001296 2556 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001299 2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001306 2556 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001310 2556 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001313 2556 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001316 2556 flags.go:64] FLAG: --pod-cidr="" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001319 2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001325 2556 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001328 2556 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001331 2556 flags.go:64] FLAG: --pods-per-core="0" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001334 2556 flags.go:64] FLAG: --port="10250" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001338 2556 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001341 2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06ac906ab954e1244" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001344 2556 flags.go:64] FLAG: --qos-reserved="" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001347 2556 flags.go:64] FLAG: --read-only-port="10255" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001383 2556 flags.go:64] FLAG: --register-node="true" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001425 2556 flags.go:64] FLAG: --register-with-taints="" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001432 2556 flags.go:64] FLAG: --registry-burst="10" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001438 2556 flags.go:64] FLAG: --registry-qps="5" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001445 2556 flags.go:64] FLAG: --reserved-cpus="" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001451 2556 flags.go:64] FLAG: --reserved-memory="" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001458 2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001464 2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 21:51:21.010520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001471 2556 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001482 2556 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001488 2556 flags.go:64] FLAG: --runonce="false" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001493 2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001499 2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001664 2556 flags.go:64] FLAG: --seccomp-default="false" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001675 2556 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001682 2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001690 2556 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001697 2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001703 2556 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001708 2556 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001715 2556 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001728 2556 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001734 2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001740 2556 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001746 2556 flags.go:64] FLAG: --system-cgroups="" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001752 2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001769 2556 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001781 2556 flags.go:64] FLAG: --tls-cert-file="" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001786 2556 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001796 2556 flags.go:64] FLAG: --tls-min-version="" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001801 2556 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001806 2556 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001811 2556 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 21:51:21.011196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001817 2556 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001822 2556 flags.go:64] FLAG: --v="2" Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001850 2556 flags.go:64] FLAG: --version="false" Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001862 2556 flags.go:64] FLAG: --vmodule="" Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001868 2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.001875 2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002286 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002296 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002303 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002308 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002315 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002321 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002327 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002334 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002340 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002344 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002355 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002360 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002370 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002431 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002546 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 21:51:21.011874 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002560 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002565 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002570 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002575 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002598 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002604 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002608 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002613 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002618 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002622 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002630 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002637 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002642 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002648 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002652 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002658 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002662 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002667 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002672 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 21:51:21.012448 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002678 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002682 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002688 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002692 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002698 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002703 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002707 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002712 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002717 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002721 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002727 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002731 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002737 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002742 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002747 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002752 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002757 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002762 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002767 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 21:51:21.012998 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002771 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002776 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002780 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002785 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002790 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002795 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002800 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002804 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002809 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002814 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002819 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002824 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002828 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002832 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002837 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002842 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002847 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002851 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002856 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002861 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 21:51:21.013492 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002865 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002870 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002874 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002879 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002884 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002888 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002894 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002899 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002903 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002908 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002913 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002918 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002923 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002929 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002935 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002940 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002945 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002950 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002955 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002959 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 21:51:21.014044 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002964 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002968 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002973 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002977 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002982 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.002986 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.003818 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.011306 2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.011321 2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.011341 2556 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.011347 2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011405 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011410 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011413 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011416 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 21:51:21.014564 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011419 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011422 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011424 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011427 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011430 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011433 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011437 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011440 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011443 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011445 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011448 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011451 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011454 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011456 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011459 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011462 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011465 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011468 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011470 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011473 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 21:51:21.015097 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011476 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011479 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011482 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011484 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011487 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011491 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011494 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011498 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011502 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011506 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011511 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011514 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011517 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011520 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011523 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011526 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011529 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011532 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011535 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 21:51:21.015618 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011538 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011541 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011543 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011546 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011549 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011551 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011554 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011557 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011560 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011562 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011565 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011568 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011571 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011574 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011576 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011579 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011582 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011584 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011588 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 21:51:21.016130 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011590 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011595 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011598 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011601 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011605 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011608 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011611 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011614 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011617 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011620 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011622 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011625 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011628 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011631 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011633 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011636 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011639 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011642 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011644 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 21:51:21.016631 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011664 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011667 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011670 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011673 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011676 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011678 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011681 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011684 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011686 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011689 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011692 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011695 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011697 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011700 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011703 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011706 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011708 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 21:51:21.017153 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011711 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.011716 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011847 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011852 2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011856 2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011859 2556 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011863 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011866 2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011869 2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011873 2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011877 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011880 2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011884 2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011887 2556 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011890 2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 21:51:21.017605 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011893 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011895 2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011898 2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011901 2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011904 2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011907 2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011909 2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011912 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011915 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011917 2556 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011920 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011923 2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011926 2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011928 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011931 2556 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011933 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011936 2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011939 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011942 2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011944 2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 21:51:21.018028 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011947 2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011950 2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011952 2556 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011955 2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011958 2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011961 2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011963 2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011966 2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011969 2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011971 2556 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011974 2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011976 2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011979 2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011982 2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011985 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011988 2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011990 2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011993 2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.011998 2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 21:51:21.018544 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012001 2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012004 2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012006 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012009 2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012012 2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012014 2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012017 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012020 2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012023 2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012025 2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012028 2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012030 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012033 2556 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012036 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012038 2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012041 2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012044 2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012047 2556 feature_gate.go:328] unrecognized feature gate: Example Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012050 2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012052 2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 21:51:21.019055 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012055 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012058 2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012061 2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012063 2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012066 2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012069 2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012071 2556 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012076 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012079 2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012082 2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012085 2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012088 2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012091 2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012094 2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012097 2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012099 2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012102 2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012105 2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012108 2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012111 2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 21:51:21.019562 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012114 2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012117 2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012120 2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012124 2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012127 2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012130 2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:21.012133 2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.012138 2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.012241 2556 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.014639 2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.016206 2556 server.go:1019] "Starting client certificate rotation" Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.016316 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.017151 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 21:51:21.020106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.019456 2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 21:51:21.023832 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.023816 2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 21:51:21.033045 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.033027 2556 log.go:25] "Validated CRI v1 runtime API" Jul 09 21:51:21.035332 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.035271 2556 log.go:25] "Validated CRI v1 image API" Jul 09 21:51:21.037174 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.037152 2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 21:51:21.043042 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.043019 2556 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8826fc8e-1c81-4a40-8d43-c521fc9e627b:/dev/nvme0n1p4 a68e8b6d-b22d-432a-aa08-896325c779d4:/dev/nvme0n1p3] Jul 09 21:51:21.043121 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.043039 2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 21:51:21.048256 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.048137 2556 manager.go:216] Machine: {Timestamp:2026-07-09 21:51:21.047006105 +0000 UTC m=+0.357623622 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101679 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fbfb534e941767bec9cf4660d1451 SystemUUID:ec2fbfb5-34e9-4176-7bec-9cf4660d1451 BootID:0a62e82f-20e1-413f-8458-83adc5cddc21 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5b:66:ba:e1:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5b:66:ba:e1:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:f0:d6:61:74:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 21:51:21.048256 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.048255 2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 21:51:21.048395 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.048382 2556 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 21:51:21.049501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.049471 2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 21:51:21.049650 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.049503 2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 21:51:21.050289 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.050278 2556 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 21:51:21.050332 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.050291 2556 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 21:51:21.050332 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.050299 2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 21:51:21.050958 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.050946 2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 21:51:21.051819 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.051800 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 21:51:21.052343 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.052331 2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 21:51:21.055380 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055368 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 09 21:51:21.055501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055492 2556 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 21:51:21.055534 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055524 2556 kubelet.go:486] "Attempting to sync node with API server" Jul 09 21:51:21.055568 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055537 2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 21:51:21.055568 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055552 2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 21:51:21.055568 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055562 2556 kubelet.go:398] "Adding apiserver pod source" Jul 09 21:51:21.055671 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055574 2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 21:51:21.055983 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055968 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 21:51:21.056069 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.055988 2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 21:51:21.059238 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.059217 2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 21:51:21.061655 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.061639 2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 21:51:21.061734 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.061689 2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 21:51:21.062969 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.062956 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 21:51:21.063014 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.062976 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 21:51:21.063014 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.062984 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 21:51:21.063014 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.062991 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 21:51:21.063014 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063004 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 21:51:21.063014 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063014 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 21:51:21.063160 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063024 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 21:51:21.063160 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063029 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 21:51:21.063160 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063036 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 21:51:21.063160 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063042 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 21:51:21.063160 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063051 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 21:51:21.063160 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063057 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 21:51:21.063922 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063910 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 21:51:21.063922 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.063922 2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 21:51:21.066030 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.066008 2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 21:51:21.066669 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.066624 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 21:51:21.066726 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.066624 2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 21:51:21.067528 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.067515 2556 server.go:1268] "Started kubelet" Jul 09 21:51:21.067644 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.067577 2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 21:51:21.067722 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.067607 2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 21:51:21.067771 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.067734 2556 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 21:51:21.068469 ip-10-0-143-160 systemd[1]: Started Kubernetes Kubelet. Jul 09 21:51:21.069256 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.069241 2556 server.go:310] "Adding debug handlers to kubelet server" Jul 09 21:51:21.072175 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.072157 2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 21:51:21.074245 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.074226 2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 21:51:21.076444 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.076429 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5xwtl" Jul 09 21:51:21.079126 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.079100 2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 21:51:21.079215 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.079121 2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 21:51:21.079792 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.079758 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.080440 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080334 2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 21:51:21.080440 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080336 2556 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 21:51:21.080440 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080364 2556 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 21:51:21.080649 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080479 2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 21:51:21.080649 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080490 2556 factory.go:55] Registering systemd factory Jul 09 21:51:21.080649 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080498 2556 factory.go:223] Registration of the systemd container factory successfully Jul 09 21:51:21.081079 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080824 2556 factory.go:153] Registering CRI-O factory Jul 09 21:51:21.081079 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080837 2556 factory.go:223] Registration of the crio container factory successfully Jul 09 21:51:21.081079 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080860 2556 factory.go:103] Registering Raw factory Jul 09 21:51:21.081079 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.080870 2556 manager.go:1196] Started watching for new ooms in manager Jul 09 21:51:21.081989 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.081970 2556 reconstruct.go:97] "Volume reconstruction finished" Jul 09 21:51:21.082102 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.082092 2556 reconciler.go:29] "Reconciler: start to sync state" Jul 09 21:51:21.082178 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.082019 2556 manager.go:349] Starting recovery of all containers Jul 09 21:51:21.082671 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.082626 2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 21:51:21.084889 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.084861 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5xwtl" Jul 09 21:51:21.087062 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.082915 2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-160.ec2.internal.18c0bd90926101ae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-160.ec2.internal,UID:ip-10-0-143-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-160.ec2.internal,},FirstTimestamp:2026-07-09 21:51:21.067487662 +0000 UTC m=+0.378105179,LastTimestamp:2026-07-09 21:51:21.067487662 +0000 UTC m=+0.378105179,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-160.ec2.internal,}" Jul 09 21:51:21.095324 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.095308 2556 manager.go:354] Recovery completed Jul 09 21:51:21.095863 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.095829 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 21:51:21.099874 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.099860 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 21:51:21.101916 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.101900 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientMemory" Jul 09 21:51:21.101992 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.101932 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 21:51:21.101992 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.101945 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientPID" Jul 09 21:51:21.102520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.102503 2556 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 21:51:21.102520 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.102519 2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 21:51:21.102669 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.102537 2556 state_mem.go:36] "Initialized new in-memory state store" Jul 09 21:51:21.102669 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.102538 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 21:51:21.103297 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.103282 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientMemory" Jul 09 21:51:21.103344 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.103310 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 21:51:21.103344 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.103322 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientPID" Jul 09 21:51:21.103762 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.103746 2556 policy_none.go:49] "None policy: Start" Jul 09 21:51:21.103762 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.103763 2556 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 21:51:21.103846 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.103775 2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 21:51:21.104825 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.104814 2556 policy_none.go:47] "Start" Jul 09 21:51:21.152224 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.152196 2556 manager.go:337] "Starting Device Plugin manager" Jul 09 21:51:21.152326 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.152244 2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 21:51:21.152326 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.152258 2556 server.go:87] "Starting device plugin registration server" Jul 09 21:51:21.152516 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.152501 2556 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 21:51:21.152571 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.152517 2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 21:51:21.152634 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.152615 2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 21:51:21.152719 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.152699 2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 21:51:21.152719 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.152714 2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 21:51:21.153338 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.153321 2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 21:51:21.153411 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.153366 2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.253664 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.253632 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 21:51:21.256463 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.256443 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientMemory" Jul 09 21:51:21.256590 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.256475 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 21:51:21.256590 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.256486 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientPID" Jul 09 21:51:21.256590 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.256514 2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.265076 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.265057 2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.265182 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.265085 2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-160.ec2.internal\": node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.267642 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.267613 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 21:51:21.269120 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.269103 2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 21:51:21.269218 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.269137 2556 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 21:51:21.269218 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.269159 2556 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 21:51:21.269218 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.269201 2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 21:51:21.271229 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.271208 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 21:51:21.285283 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.285260 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.370840 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.370767 2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal"] Jul 09 21:51:21.370932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.370845 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 21:51:21.372849 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.372829 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientMemory" Jul 09 21:51:21.372957 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.372870 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 21:51:21.372957 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.372888 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientPID" Jul 09 21:51:21.374263 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.374247 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 21:51:21.374399 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.374384 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.374451 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.374416 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 21:51:21.375833 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.375353 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientMemory" Jul 09 21:51:21.375833 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.375396 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 21:51:21.375833 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.375414 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientPID" Jul 09 21:51:21.375833 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.375765 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientMemory" Jul 09 21:51:21.375833 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.375783 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 21:51:21.375833 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.375795 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientPID" Jul 09 21:51:21.377062 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.377045 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.377154 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.377070 2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 21:51:21.377798 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.377779 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientMemory" Jul 09 21:51:21.377888 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.377813 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 21:51:21.377888 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.377829 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeHasSufficientPID" Jul 09 21:51:21.384081 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.384062 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fd8be6086eaa831942ea0c0f68aec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal\" (UID: \"00fd8be6086eaa831942ea0c0f68aec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.384174 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.384089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fd8be6086eaa831942ea0c0f68aec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal\" (UID: \"00fd8be6086eaa831942ea0c0f68aec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.384174 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.384107 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/487f50e765b2ad11dd8e605acdc1e631-config\") pod \"kube-apiserver-proxy-ip-10-0-143-160.ec2.internal\" (UID: \"487f50e765b2ad11dd8e605acdc1e631\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.386242 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.386220 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.404413 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.404393 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-160.ec2.internal\" not found" node="ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.408940 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.408922 2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-160.ec2.internal\" not found" node="ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.484769 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.484730 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fd8be6086eaa831942ea0c0f68aec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal\" (UID: \"00fd8be6086eaa831942ea0c0f68aec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.484769 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.484766 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fd8be6086eaa831942ea0c0f68aec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal\" (UID: \"00fd8be6086eaa831942ea0c0f68aec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.485011 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.484784 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/487f50e765b2ad11dd8e605acdc1e631-config\") pod \"kube-apiserver-proxy-ip-10-0-143-160.ec2.internal\" (UID: \"487f50e765b2ad11dd8e605acdc1e631\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.485011 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.484800 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00fd8be6086eaa831942ea0c0f68aec6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal\" (UID: \"00fd8be6086eaa831942ea0c0f68aec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.485011 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.484820 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/487f50e765b2ad11dd8e605acdc1e631-config\") pod \"kube-apiserver-proxy-ip-10-0-143-160.ec2.internal\" (UID: \"487f50e765b2ad11dd8e605acdc1e631\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.485011 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.484832 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00fd8be6086eaa831942ea0c0f68aec6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal\" (UID: \"00fd8be6086eaa831942ea0c0f68aec6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.486996 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.486978 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.587730 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.587702 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.688774 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.688694 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.708314 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.708292 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.713514 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:21.713497 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" Jul 09 21:51:21.788803 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.788761 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.889350 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.889313 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:21.990305 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:21.990229 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:22.015337 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.015306 2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 21:51:22.015484 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.015465 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 21:51:22.015534 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.015510 2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 21:51:22.087868 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.087832 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 21:46:21 +0000 UTC" deadline="2028-04-09 04:55:19.583974626 +0000 UTC" Jul 09 21:51:22.087868 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.087857 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15343h3m57.496119818s" Jul 09 21:51:22.091117 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:22.091103 2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-160.ec2.internal\" not found" Jul 09 21:51:22.122625 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.122579 2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 21:51:22.179943 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.179914 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" Jul 09 21:51:22.192265 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.192232 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 21:51:22.194648 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.194627 2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" Jul 09 21:51:22.203075 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.203057 2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 21:51:22.266847 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:22.266814 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487f50e765b2ad11dd8e605acdc1e631.slice/crio-5efcc2ab264913c39bdc55e87389f0e306e07ba1bf3ef9d62d8051804745e31a WatchSource:0}: Error finding container 5efcc2ab264913c39bdc55e87389f0e306e07ba1bf3ef9d62d8051804745e31a: Status 404 returned error can't find the container with id 5efcc2ab264913c39bdc55e87389f0e306e07ba1bf3ef9d62d8051804745e31a Jul 09 21:51:22.267190 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:22.267156 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fd8be6086eaa831942ea0c0f68aec6.slice/crio-5421e14568817f60ba22c87ccd33cb905f69e942c857e3c081b43e20ee6d92e7 WatchSource:0}: Error finding container 5421e14568817f60ba22c87ccd33cb905f69e942c857e3c081b43e20ee6d92e7: Status 404 returned error can't find the container with id 5421e14568817f60ba22c87ccd33cb905f69e942c857e3c081b43e20ee6d92e7 Jul 09 21:51:22.272411 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.272392 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 21:51:22.272629 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.272562 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" event={"ID":"487f50e765b2ad11dd8e605acdc1e631","Type":"ContainerStarted","Data":"5efcc2ab264913c39bdc55e87389f0e306e07ba1bf3ef9d62d8051804745e31a"} Jul 09 21:51:22.273634 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.273611 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" event={"ID":"00fd8be6086eaa831942ea0c0f68aec6","Type":"ContainerStarted","Data":"5421e14568817f60ba22c87ccd33cb905f69e942c857e3c081b43e20ee6d92e7"} Jul 09 21:51:22.579604 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:22.579510 2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 21:51:23.056803 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.056715 2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 21:51:23.057388 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.056812 2556 apiserver.go:52] "Watching apiserver" Jul 09 21:51:23.064937 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.064908 2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 21:51:23.065438 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.065399 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-5jcvb","openshift-network-diagnostics/network-check-target-k9bcs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk","openshift-dns/node-resolver-9znf9","openshift-image-registry/node-ca-dp58g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal","openshift-network-operator/iptables-alerter-kghj8","openshift-ovn-kubernetes/ovnkube-node-dmq54","kube-system/global-pull-secret-syncer-bjk4m","kube-system/konnectivity-agent-fvpk9","kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal","openshift-cluster-node-tuning-operator/tuned-58pcd","openshift-multus/multus-additional-cni-plugins-hzrbj","openshift-multus/multus-bntnp"] Jul 09 21:51:23.066967 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.066949 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.068216 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.068155 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.069700 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.069666 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 21:51:23.070074 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.069907 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 21:51:23.070074 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.069976 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 21:51:23.070074 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.069994 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sg5g\"" Jul 09 21:51:23.070074 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.069997 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.071050 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.070996 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.071890 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.071858 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 21:51:23.071989 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.071954 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 21:51:23.072125 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.072105 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 21:51:23.072217 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.072200 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k4bhg\"" Jul 09 21:51:23.072929 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.072583 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 21:51:23.072929 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.072751 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 21:51:23.072929 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.072828 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mzb2r\"" Jul 09 21:51:23.073141 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.072962 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 21:51:23.073141 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.073036 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 21:51:23.073247 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.073235 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 21:51:23.073954 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.073931 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h5925\"" Jul 09 21:51:23.074290 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.074273 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 21:51:23.074556 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.074541 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 21:51:23.074867 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.074845 2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 21:51:23.076555 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.075317 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.078095 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.078064 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 21:51:23.078095 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.078070 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 21:51:23.078980 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.078956 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4p4v6\"" Jul 09 21:51:23.079073 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.079006 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 21:51:23.079667 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.079479 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.080821 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.080800 2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 21:51:23.082079 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.081625 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.082079 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.081750 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:23.082245 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.082086 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 21:51:23.082935 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.082916 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:23.083034 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.083000 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:23.083462 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.083442 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fpzfs\"" Jul 09 21:51:23.084331 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.084312 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:23.084422 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.084400 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:23.085759 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.085739 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:23.088221 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.088203 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.088907 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.088885 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.090113 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.090093 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 21:51:23.090527 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.090508 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9vgl5\"" Jul 09 21:51:23.091120 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.091102 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 21:51:23.091484 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.091464 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 21:51:23.091552 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.091512 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wtdk7\"" Jul 09 21:51:23.092148 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.092128 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 21:51:23.092302 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.092286 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5w9t4\"" Jul 09 21:51:23.092412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.092389 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 21:51:23.092412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.092401 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 21:51:23.092412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.092349 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 21:51:23.092707 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.092404 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 21:51:23.092707 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.092359 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 21:51:23.092707 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.092327 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 21:51:23.093204 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.093179 2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 21:51:23.094724 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.094703 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.095112 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.095094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b9736d4-bfff-4aa5-a935-f5691adee10f-host-slash\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.095247 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.095231 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxtp\" (UniqueName: \"kubernetes.io/projected/8b9736d4-bfff-4aa5-a935-f5691adee10f-kube-api-access-gbxtp\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.095355 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.095342 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-system-cni-dir\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.095451 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.095435 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-os-release\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.095563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.095543 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-device-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.095683 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.095644 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-system-cni-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.095916 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.095900 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-hostroot\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.096033 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.096016 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.096182 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.096153 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-os-release\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.096896 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.096878 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-etc-kubernetes\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.097328 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.097309 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8rv\" (UniqueName: \"kubernetes.io/projected/fd78623e-cb49-4f50-ae98-3eb77e2c762e-kube-api-access-cb8rv\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.097518 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.097501 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2e19c4a-e0b3-46a7-b8de-563c6b248018-kubelet-config\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.097667 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.097636 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.097794 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.097778 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.097907 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.097892 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcsh7\" (UniqueName: \"kubernetes.io/projected/0d9060a4-7039-4b8a-a3af-118f7aa90010-kube-api-access-tcsh7\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.098040 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.098026 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-hosts-file\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.098140 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.098127 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdzz\" (UniqueName: \"kubernetes.io/projected/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-kube-api-access-kjdzz\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.098254 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.098234 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-daemon-config\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.098706 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.098686 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-multus-certs\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.098856 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.098838 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9m2k\" (UniqueName: \"kubernetes.io/projected/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-kube-api-access-q9m2k\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:23.099009 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.098989 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.099109 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099021 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.099109 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099045 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-sys-fs\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.099109 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099067 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-socket-dir-parent\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.099279 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099111 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-cni-bin\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.099279 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099136 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-host\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.099279 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099164 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqq9m\" (UniqueName: \"kubernetes.io/projected/82a91008-923a-4f71-9420-b743b9a391e9-kube-api-access-dqq9m\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.099279 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-socket-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.099279 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099224 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-registration-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.099279 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099250 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-cni-multus\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099275 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2e19c4a-e0b3-46a7-b8de-563c6b248018-dbus\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099303 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-cnibin\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099329 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-conf-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099364 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-serviceca\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099388 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdckx\" (UniqueName: \"kubernetes.io/projected/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-kube-api-access-tdckx\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099424 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-cnibin\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099451 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-tmp-dir\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099485 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-cni-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099517 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd78623e-cb49-4f50-ae98-3eb77e2c762e-cni-binary-copy\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099541 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-k8s-cni-cncf-io\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.099582 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099566 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-kubelet\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.100208 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099704 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:23.100208 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099756 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b9736d4-bfff-4aa5-a935-f5691adee10f-iptables-alerter-script\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.100208 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099801 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.100208 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.099834 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-netns\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.120208 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.120189 2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xl7gs" Jul 09 21:51:23.130000 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.129941 2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xl7gs" Jul 09 21:51:23.182154 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.182128 2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 21:51:23.200061 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200025 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-cni-bin\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.200242 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200070 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-host\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.200242 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200105 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-systemd\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.200242 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200161 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-cni-bin\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.200242 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200168 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-host\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.200242 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200229 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-registration-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200272 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2e19c4a-e0b3-46a7-b8de-563c6b248018-dbus\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200301 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-cnibin\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200337 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-registration-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200383 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-cnibin\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200382 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-conf-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200428 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-conf-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200428 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-serviceca\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200475 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdckx\" (UniqueName: \"kubernetes.io/projected/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-kube-api-access-tdckx\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200506 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-var-lib-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200514 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a2e19c4a-e0b3-46a7-b8de-563c6b248018-dbus\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200536 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.200563 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200562 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-cni-bin\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200586 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-sys\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200635 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-cnibin\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200663 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-cni-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200690 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd78623e-cb49-4f50-ae98-3eb77e2c762e-cni-binary-copy\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200703 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-cnibin\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200758 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-k8s-cni-cncf-io\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-k8s-cni-cncf-io\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200772 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-cni-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200803 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200829 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c4578ac-8173-4dbc-8332-250b5a40fd7a-konnectivity-ca\") pod \"konnectivity-agent-fvpk9\" (UID: \"1c4578ac-8173-4dbc-8332-250b5a40fd7a\") " pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200848 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-node-log\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200866 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200896 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b9736d4-bfff-4aa5-a935-f5691adee10f-iptables-alerter-script\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200921 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-systemd-units\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200971 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-log-socket\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.200984 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.200987 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-serviceca\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.201299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.201031 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.202151 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.201065 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:23.701043551 +0000 UTC m=+3.011661061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:23.202151 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.201113 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-ovnkube-config\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.202151 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202033 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b9736d4-bfff-4aa5-a935-f5691adee10f-iptables-alerter-script\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.202329 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202309 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd78623e-cb49-4f50-ae98-3eb77e2c762e-cni-binary-copy\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.202384 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.201139 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-env-overrides\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.202438 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202391 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb908ed8-23f3-496e-a282-218bd4208b2d-ovn-node-metrics-cert\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.202487 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202433 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxtp\" (UniqueName: \"kubernetes.io/projected/8b9736d4-bfff-4aa5-a935-f5691adee10f-kube-api-access-gbxtp\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.202487 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202466 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-system-cni-dir\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.202571 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202493 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-device-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.202571 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202524 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-system-cni-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.202571 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202555 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-systemd\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.202706 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202588 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-lib-modules\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.202706 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202619 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ccaed5-cce9-46ff-9865-09829405516f-tmp\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.202706 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202644 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2v6\" (UniqueName: \"kubernetes.io/projected/19ccaed5-cce9-46ff-9865-09829405516f-kube-api-access-4c2v6\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.202706 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202694 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.202865 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202728 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-os-release\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.202865 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8rv\" (UniqueName: \"kubernetes.io/projected/fd78623e-cb49-4f50-ae98-3eb77e2c762e-kube-api-access-cb8rv\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.202865 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202792 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-kubelet\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.202865 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202820 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-ovnkube-script-lib\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.203004 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202867 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5s9v\" (UniqueName: \"kubernetes.io/projected/bb908ed8-23f3-496e-a282-218bd4208b2d-kube-api-access-m5s9v\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.203004 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202899 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.203004 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202931 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.203004 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202962 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-hosts-file\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.203004 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.202995 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdzz\" (UniqueName: \"kubernetes.io/projected/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-kube-api-access-kjdzz\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.203241 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203028 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-daemon-config\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.203241 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203063 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-multus-certs\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.203241 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-cni-netd\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.203241 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203122 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.203241 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203207 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-os-release\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.203501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203256 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.203501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203293 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-sys-fs\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.203501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203341 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-socket-dir-parent\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.203501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203379 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-ovn\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.203501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203412 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqq9m\" (UniqueName: \"kubernetes.io/projected/82a91008-923a-4f71-9420-b743b9a391e9-kube-api-access-dqq9m\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.203501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203447 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-socket-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.203501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203481 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-cni-multus\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203518 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203554 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-modprobe-d\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203583 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-run\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203631 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-var-lib-kubelet\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203681 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-run-netns\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203700 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203716 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-kubernetes\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203753 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-tmp-dir\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.203873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203794 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-kubelet\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.204264 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.203952 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-socket-dir-parent\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.204264 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204030 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-multus-certs\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.204264 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204072 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-system-cni-dir\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.204264 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204096 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-tmp-dir\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.204264 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204183 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysconfig\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.204264 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204217 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-host\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.204526 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.204526 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204510 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-netns\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.204596 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204585 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-sys-fs\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204682 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204699 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-kubelet\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204725 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-slash\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204759 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b9736d4-bfff-4aa5-a935-f5691adee10f-host-slash\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204797 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-os-release\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204801 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204835 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-hostroot\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.204957 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205078 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fd78623e-cb49-4f50-ae98-3eb77e2c762e-multus-daemon-config\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205318 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-hosts-file\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205296 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c4578ac-8173-4dbc-8332-250b5a40fd7a-agent-certs\") pod \"konnectivity-agent-fvpk9\" (UID: \"1c4578ac-8173-4dbc-8332-250b5a40fd7a\") " pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205354 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b9736d4-bfff-4aa5-a935-f5691adee10f-host-slash\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205359 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-hostroot\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205377 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-etc-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205407 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysctl-conf\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205441 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ccaed5-cce9-46ff-9865-09829405516f-etc-tuned\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.205452 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:23.206550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205504 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-run-netns\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205519 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205466 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205408 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-host-var-lib-cni-multus\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205614 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-device-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205605 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0d9060a4-7039-4b8a-a3af-118f7aa90010-socket-dir\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205633 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-etc-kubernetes\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205707 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/82a91008-923a-4f71-9420-b743b9a391e9-os-release\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.205726 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret podName:a2e19c4a-e0b3-46a7-b8de-563c6b248018 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:23.705695496 +0000 UTC m=+3.016313004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret") pod "global-pull-secret-syncer-bjk4m" (UID: "a2e19c4a-e0b3-46a7-b8de-563c6b248018") : object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205748 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-system-cni-dir\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2e19c4a-e0b3-46a7-b8de-563c6b248018-kubelet-config\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205904 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysctl-d\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.205963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcsh7\" (UniqueName: \"kubernetes.io/projected/0d9060a4-7039-4b8a-a3af-118f7aa90010-kube-api-access-tcsh7\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.206000 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9m2k\" (UniqueName: \"kubernetes.io/projected/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-kube-api-access-q9m2k\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.206095 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd78623e-cb49-4f50-ae98-3eb77e2c762e-etc-kubernetes\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.206234 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a2e19c4a-e0b3-46a7-b8de-563c6b248018-kubelet-config\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.207551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.206246 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/82a91008-923a-4f71-9420-b743b9a391e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.222136 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.222108 2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 21:51:23.226882 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.226678 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdzz\" (UniqueName: \"kubernetes.io/projected/a2ca5280-d3ac-4cec-82ed-67dcdfff85f8-kube-api-access-kjdzz\") pod \"node-resolver-9znf9\" (UID: \"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8\") " pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.227597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.227550 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdckx\" (UniqueName: \"kubernetes.io/projected/95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b-kube-api-access-tdckx\") pod \"node-ca-dp58g\" (UID: \"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b\") " pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.228099 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.228072 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqq9m\" (UniqueName: \"kubernetes.io/projected/82a91008-923a-4f71-9420-b743b9a391e9-kube-api-access-dqq9m\") pod \"multus-additional-cni-plugins-hzrbj\" (UID: \"82a91008-923a-4f71-9420-b743b9a391e9\") " pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.228619 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.228592 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8rv\" (UniqueName: \"kubernetes.io/projected/fd78623e-cb49-4f50-ae98-3eb77e2c762e-kube-api-access-cb8rv\") pod \"multus-bntnp\" (UID: \"fd78623e-cb49-4f50-ae98-3eb77e2c762e\") " pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.228746 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.228593 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxtp\" (UniqueName: \"kubernetes.io/projected/8b9736d4-bfff-4aa5-a935-f5691adee10f-kube-api-access-gbxtp\") pod \"iptables-alerter-kghj8\" (UID: \"8b9736d4-bfff-4aa5-a935-f5691adee10f\") " pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.230645 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.230598 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9m2k\" (UniqueName: \"kubernetes.io/projected/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-kube-api-access-q9m2k\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:23.230745 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.230726 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcsh7\" (UniqueName: \"kubernetes.io/projected/0d9060a4-7039-4b8a-a3af-118f7aa90010-kube-api-access-tcsh7\") pod \"aws-ebs-csi-driver-node-8lsjk\" (UID: \"0d9060a4-7039-4b8a-a3af-118f7aa90010\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.306662 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306620 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-slash\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.306662 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306667 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c4578ac-8173-4dbc-8332-250b5a40fd7a-agent-certs\") pod \"konnectivity-agent-fvpk9\" (UID: \"1c4578ac-8173-4dbc-8332-250b5a40fd7a\") " pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306692 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-etc-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysctl-conf\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306736 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ccaed5-cce9-46ff-9865-09829405516f-etc-tuned\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306751 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-slash\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306759 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysctl-d\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306825 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-systemd\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306858 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-var-lib-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306886 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306891 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-etc-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306910 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysctl-d\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306911 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-cni-bin\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.306940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306946 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-cni-bin\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306955 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-sys\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.306984 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-systemd\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307012 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c4578ac-8173-4dbc-8332-250b5a40fd7a-konnectivity-ca\") pod \"konnectivity-agent-fvpk9\" (UID: \"1c4578ac-8173-4dbc-8332-250b5a40fd7a\") " pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307016 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysctl-conf\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307026 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307040 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-node-log\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307065 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-sys\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307067 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307079 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-var-lib-openvswitch\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307105 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-systemd-units\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307110 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-run-ovn-kubernetes\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307142 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-log-socket\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307146 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-systemd-units\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307171 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307186 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-node-log\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307230 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-ovnkube-config\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307260 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-env-overrides\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.307574 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307265 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-log-socket\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307286 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb908ed8-23f3-496e-a282-218bd4208b2d-ovn-node-metrics-cert\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307316 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-systemd\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307340 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-lib-modules\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307357 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307366 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ccaed5-cce9-46ff-9865-09829405516f-tmp\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307419 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2v6\" (UniqueName: \"kubernetes.io/projected/19ccaed5-cce9-46ff-9865-09829405516f-kube-api-access-4c2v6\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-kubelet\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307485 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-ovnkube-script-lib\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307510 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5s9v\" (UniqueName: \"kubernetes.io/projected/bb908ed8-23f3-496e-a282-218bd4208b2d-kube-api-access-m5s9v\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307546 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-cni-netd\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307576 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-ovn\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307623 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307649 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-modprobe-d\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-run\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307683 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c4578ac-8173-4dbc-8332-250b5a40fd7a-konnectivity-ca\") pod \"konnectivity-agent-fvpk9\" (UID: \"1c4578ac-8173-4dbc-8332-250b5a40fd7a\") " pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307701 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-var-lib-kubelet\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.308597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307743 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-run-netns\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307764 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-var-lib-kubelet\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307792 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-kubernetes\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysconfig\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307846 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-host\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307898 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-ovnkube-config\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307925 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-host\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307968 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-run-netns\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.307982 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-kubelet\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308005 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-host-cni-netd\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308059 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-sysconfig\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308066 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb908ed8-23f3-496e-a282-218bd4208b2d-run-ovn\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308121 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-systemd\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308134 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-modprobe-d\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308185 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-lib-modules\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308199 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-run\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308242 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-env-overrides\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308518 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19ccaed5-cce9-46ff-9865-09829405516f-etc-kubernetes\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.309550 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.308696 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb908ed8-23f3-496e-a282-218bd4208b2d-ovnkube-script-lib\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.310443 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.309900 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19ccaed5-cce9-46ff-9865-09829405516f-etc-tuned\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.310443 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.309954 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c4578ac-8173-4dbc-8332-250b5a40fd7a-agent-certs\") pod \"konnectivity-agent-fvpk9\" (UID: \"1c4578ac-8173-4dbc-8332-250b5a40fd7a\") " pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:23.310443 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.309956 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19ccaed5-cce9-46ff-9865-09829405516f-tmp\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.311040 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.311012 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb908ed8-23f3-496e-a282-218bd4208b2d-ovn-node-metrics-cert\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.316536 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.316454 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 21:51:23.316674 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.316549 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 21:51:23.316674 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.316565 2556 projected.go:196] Error preparing data for projected volume kube-api-access-b4kvj for pod openshift-network-diagnostics/network-check-target-k9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:23.316674 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.316653 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj podName:aa94d66c-34f4-40a5-b073-f439fe7f01ed nodeName:}" failed. No retries permitted until 2026-07-09 21:51:23.816634917 +0000 UTC m=+3.127252434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b4kvj" (UniqueName: "kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj") pod "network-check-target-k9bcs" (UID: "aa94d66c-34f4-40a5-b073-f439fe7f01ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:23.319118 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.319092 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5s9v\" (UniqueName: \"kubernetes.io/projected/bb908ed8-23f3-496e-a282-218bd4208b2d-kube-api-access-m5s9v\") pod \"ovnkube-node-dmq54\" (UID: \"bb908ed8-23f3-496e-a282-218bd4208b2d\") " pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.319493 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.319402 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2v6\" (UniqueName: \"kubernetes.io/projected/19ccaed5-cce9-46ff-9865-09829405516f-kube-api-access-4c2v6\") pod \"tuned-58pcd\" (UID: \"19ccaed5-cce9-46ff-9865-09829405516f\") " pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.382999 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.382959 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kghj8" Jul 09 21:51:23.394135 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.394106 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" Jul 09 21:51:23.402345 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.402316 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" Jul 09 21:51:23.408666 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.408640 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9znf9" Jul 09 21:51:23.416197 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.416180 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dp58g" Jul 09 21:51:23.424017 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.423992 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bntnp" Jul 09 21:51:23.432902 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.432882 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:23.441800 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.441779 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:23.455700 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.455680 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-58pcd" Jul 09 21:51:23.711465 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.711383 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:23.711465 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.711429 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:23.711681 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.711598 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:23.711681 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.711613 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:23.711681 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.711662 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:24.711642096 +0000 UTC m=+4.022259604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:23.711811 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.711700 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret podName:a2e19c4a-e0b3-46a7-b8de-563c6b248018 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:24.711680225 +0000 UTC m=+4.022297745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret") pod "global-pull-secret-syncer-bjk4m" (UID: "a2e19c4a-e0b3-46a7-b8de-563c6b248018") : object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:23.913076 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:23.913024 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:23.913269 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.913238 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 21:51:23.913323 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.913266 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 21:51:23.913323 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.913296 2556 projected.go:196] Error preparing data for projected volume kube-api-access-b4kvj for pod openshift-network-diagnostics/network-check-target-k9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:23.913402 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:23.913358 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj podName:aa94d66c-34f4-40a5-b073-f439fe7f01ed nodeName:}" failed. No retries permitted until 2026-07-09 21:51:24.913342978 +0000 UTC m=+4.223960482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4kvj" (UniqueName: "kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj") pod "network-check-target-k9bcs" (UID: "aa94d66c-34f4-40a5-b073-f439fe7f01ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:24.133635 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:24.133537 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 21:46:23 +0000 UTC" deadline="2028-01-04 00:59:21.662482654 +0000 UTC" Jul 09 21:51:24.133635 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:24.133615 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13035h7m57.528874213s" Jul 09 21:51:24.409597 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:24.409564 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ca5280_d3ac_4cec_82ed_67dcdfff85f8.slice/crio-13fca0df9fe633336367296aab04e3e778f418558685a43c3bc0a60033b4c244 WatchSource:0}: Error finding container 13fca0df9fe633336367296aab04e3e778f418558685a43c3bc0a60033b4c244: Status 404 returned error can't find the container with id 13fca0df9fe633336367296aab04e3e778f418558685a43c3bc0a60033b4c244 Jul 09 21:51:24.438260 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:24.438226 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9736d4_bfff_4aa5_a935_f5691adee10f.slice/crio-586c73a8424ca9fd04062c16fcfe0ccfb2c3bb3f7fdaac3e1839b0bac72d34bf WatchSource:0}: Error finding container 586c73a8424ca9fd04062c16fcfe0ccfb2c3bb3f7fdaac3e1839b0bac72d34bf: Status 404 returned error can't find the container with id 586c73a8424ca9fd04062c16fcfe0ccfb2c3bb3f7fdaac3e1839b0bac72d34bf Jul 09 21:51:24.439080 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:24.439015 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb908ed8_23f3_496e_a282_218bd4208b2d.slice/crio-fcd5337d769ad86d3f702253160d2f82f96fd6e5f40dcd9f1502447f0db3140a WatchSource:0}: Error finding container fcd5337d769ad86d3f702253160d2f82f96fd6e5f40dcd9f1502447f0db3140a: Status 404 returned error can't find the container with id fcd5337d769ad86d3f702253160d2f82f96fd6e5f40dcd9f1502447f0db3140a Jul 09 21:51:24.440954 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:24.440932 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9060a4_7039_4b8a_a3af_118f7aa90010.slice/crio-f8553f7ba0e69655cae63dc2977d348cd1466ed7bbe0f1eb99408a3659561bca WatchSource:0}: Error finding container f8553f7ba0e69655cae63dc2977d348cd1466ed7bbe0f1eb99408a3659561bca: Status 404 returned error can't find the container with id f8553f7ba0e69655cae63dc2977d348cd1466ed7bbe0f1eb99408a3659561bca Jul 09 21:51:24.721115 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:24.720936 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:24.721202 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:24.721161 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:24.721202 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:24.721085 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:24.721287 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:24.721254 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:24.721287 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:24.721262 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:26.721241062 +0000 UTC m=+6.031858567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:24.721373 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:24.721292 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret podName:a2e19c4a-e0b3-46a7-b8de-563c6b248018 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:26.721280369 +0000 UTC m=+6.031897873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret") pod "global-pull-secret-syncer-bjk4m" (UID: "a2e19c4a-e0b3-46a7-b8de-563c6b248018") : object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:24.922524 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:24.921769 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:24.922524 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:24.922009 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 21:51:24.922524 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:24.922031 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 21:51:24.922524 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:24.922044 2556 projected.go:196] Error preparing data for projected volume kube-api-access-b4kvj for pod openshift-network-diagnostics/network-check-target-k9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:24.922524 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:24.922107 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj podName:aa94d66c-34f4-40a5-b073-f439fe7f01ed nodeName:}" failed. No retries permitted until 2026-07-09 21:51:26.922088042 +0000 UTC m=+6.232705548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4kvj" (UniqueName: "kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj") pod "network-check-target-k9bcs" (UID: "aa94d66c-34f4-40a5-b073-f439fe7f01ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:25.133985 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.133889 2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 21:46:23 +0000 UTC" deadline="2028-01-01 19:37:45.669784432 +0000 UTC" Jul 09 21:51:25.133985 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.133952 2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12981h46m20.535836873s" Jul 09 21:51:25.270707 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.270298 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:25.270707 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.270298 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:25.270707 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:25.270446 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:25.270707 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:25.270484 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:25.270707 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.270531 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:25.270707 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:25.270634 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:25.333641 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.333578 2556 generic.go:356] "Generic (PLEG): container finished" podID="00fd8be6086eaa831942ea0c0f68aec6" containerID="b11fce148c63755c3634624f3232ae38cf5a886c4acbb01a90e772afb2a51859" exitCode=0 Jul 09 21:51:25.333835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.333746 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" event={"ID":"00fd8be6086eaa831942ea0c0f68aec6","Type":"ContainerDied","Data":"b11fce148c63755c3634624f3232ae38cf5a886c4acbb01a90e772afb2a51859"} Jul 09 21:51:25.339902 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.339849 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bntnp" event={"ID":"fd78623e-cb49-4f50-ae98-3eb77e2c762e","Type":"ContainerStarted","Data":"811cc37a4c81cacec5cf1eab03c534103f0585b65e4e3ffbe25aa169f868aa5b"} Jul 09 21:51:25.350788 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.350755 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-58pcd" event={"ID":"19ccaed5-cce9-46ff-9865-09829405516f","Type":"ContainerStarted","Data":"fc879ec07da2ef71f8ed2128906f7938b8bb484f612cc92b2c70fd98f3aabb0d"} Jul 09 21:51:25.364587 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.364510 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dp58g" event={"ID":"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b","Type":"ContainerStarted","Data":"f66533871ad288e170a7215c350aa47731a6fe074a8872f99f52304c28b99586"} Jul 09 21:51:25.373013 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.372983 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" event={"ID":"82a91008-923a-4f71-9420-b743b9a391e9","Type":"ContainerStarted","Data":"5b1e978262c807b9284728ffbabcb32399bf28ae680b82b79ac704fbb6435b98"} Jul 09 21:51:25.387448 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.387394 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9znf9" event={"ID":"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8","Type":"ContainerStarted","Data":"13fca0df9fe633336367296aab04e3e778f418558685a43c3bc0a60033b4c244"} Jul 09 21:51:25.400073 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.400041 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" event={"ID":"487f50e765b2ad11dd8e605acdc1e631","Type":"ContainerStarted","Data":"05a288b145cc178874ee434d10f1049ac51f274d7a2a11fe7822757ffddc6aaf"} Jul 09 21:51:25.412332 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.412300 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" event={"ID":"0d9060a4-7039-4b8a-a3af-118f7aa90010","Type":"ContainerStarted","Data":"f8553f7ba0e69655cae63dc2977d348cd1466ed7bbe0f1eb99408a3659561bca"} Jul 09 21:51:25.436003 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.435875 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"fcd5337d769ad86d3f702253160d2f82f96fd6e5f40dcd9f1502447f0db3140a"} Jul 09 21:51:25.439975 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.439896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fvpk9" event={"ID":"1c4578ac-8173-4dbc-8332-250b5a40fd7a","Type":"ContainerStarted","Data":"d48b40010cbdd9fbe4de1d0508111fe3f1c1be594f59f567ac64d695d19ffd57"} Jul 09 21:51:25.461106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:25.461069 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kghj8" event={"ID":"8b9736d4-bfff-4aa5-a935-f5691adee10f","Type":"ContainerStarted","Data":"586c73a8424ca9fd04062c16fcfe0ccfb2c3bb3f7fdaac3e1839b0bac72d34bf"} Jul 09 21:51:26.496012 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:26.495937 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" event={"ID":"00fd8be6086eaa831942ea0c0f68aec6","Type":"ContainerStarted","Data":"e49509eb35d3cda41968583f638d01853614deb75f74961931055878fa603af0"} Jul 09 21:51:26.511743 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:26.510846 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-160.ec2.internal" podStartSLOduration=4.51082333 podStartE2EDuration="4.51082333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:26.510639454 +0000 UTC m=+5.821256982" watchObservedRunningTime="2026-07-09 21:51:26.51082333 +0000 UTC m=+5.821440858" Jul 09 21:51:26.511743 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:26.511293 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-160.ec2.internal" podStartSLOduration=4.511285613 podStartE2EDuration="4.511285613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:25.421473339 +0000 UTC m=+4.732090865" watchObservedRunningTime="2026-07-09 21:51:26.511285613 +0000 UTC m=+5.821903152" Jul 09 21:51:26.739555 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:26.738620 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:26.739555 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:26.738716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:26.739555 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:26.738947 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:26.739555 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:26.739017 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:30.738996096 +0000 UTC m=+10.049613601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:26.739555 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:26.739467 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:26.739555 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:26.739519 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret podName:a2e19c4a-e0b3-46a7-b8de-563c6b248018 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:30.739503179 +0000 UTC m=+10.050120682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret") pod "global-pull-secret-syncer-bjk4m" (UID: "a2e19c4a-e0b3-46a7-b8de-563c6b248018") : object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:26.940549 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:26.940453 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:26.940786 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:26.940685 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 21:51:26.940786 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:26.940710 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 21:51:26.940786 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:26.940724 2556 projected.go:196] Error preparing data for projected volume kube-api-access-b4kvj for pod openshift-network-diagnostics/network-check-target-k9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:26.940786 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:26.940788 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj podName:aa94d66c-34f4-40a5-b073-f439fe7f01ed nodeName:}" failed. No retries permitted until 2026-07-09 21:51:30.940766941 +0000 UTC m=+10.251384448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4kvj" (UniqueName: "kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj") pod "network-check-target-k9bcs" (UID: "aa94d66c-34f4-40a5-b073-f439fe7f01ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:27.270758 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:27.270673 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:27.270943 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:27.270831 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:27.271288 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:27.271270 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:27.271388 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:27.271362 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:27.271496 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:27.271473 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:27.271639 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:27.271601 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:29.270218 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:29.270143 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:29.270218 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:29.270196 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:29.270748 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:29.270143 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:29.270748 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:29.270289 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:29.270748 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:29.270389 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:29.270748 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:29.270499 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:30.773338 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:30.773302 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:30.773803 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:30.773365 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:30.773803 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:30.773466 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:30.773803 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:30.773468 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:30.773803 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:30.773522 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:38.773506488 +0000 UTC m=+18.084123999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:30.773803 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:30.773546 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret podName:a2e19c4a-e0b3-46a7-b8de-563c6b248018 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:38.77353044 +0000 UTC m=+18.084147942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret") pod "global-pull-secret-syncer-bjk4m" (UID: "a2e19c4a-e0b3-46a7-b8de-563c6b248018") : object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:30.975168 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:30.975133 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:30.975351 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:30.975334 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 21:51:30.975426 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:30.975358 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 21:51:30.975426 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:30.975371 2556 projected.go:196] Error preparing data for projected volume kube-api-access-b4kvj for pod openshift-network-diagnostics/network-check-target-k9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:30.975525 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:30.975437 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj podName:aa94d66c-34f4-40a5-b073-f439fe7f01ed nodeName:}" failed. No retries permitted until 2026-07-09 21:51:38.975416645 +0000 UTC m=+18.286034159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4kvj" (UniqueName: "kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj") pod "network-check-target-k9bcs" (UID: "aa94d66c-34f4-40a5-b073-f439fe7f01ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:31.278588 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:31.277044 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:31.278588 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:31.277217 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:31.278588 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:31.277672 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:31.278588 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:31.277793 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:31.278588 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:31.277849 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:31.278588 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:31.277950 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:33.269712 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:33.269679 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:33.270179 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:33.269688 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:33.270179 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:33.269819 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:33.270179 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:33.269689 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:33.270179 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:33.269916 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:33.270179 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:33.269940 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:35.269992 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:35.269907 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:35.270392 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:35.269907 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:35.270392 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:35.270022 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:35.270392 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:35.269907 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:35.270392 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:35.270078 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:35.270392 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:35.270198 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:37.269524 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:37.269489 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:37.269983 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:37.269505 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:37.269983 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:37.269497 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:37.269983 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:37.269674 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:37.269983 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:37.269761 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:37.269983 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:37.269851 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:38.838262 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:38.838227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:38.838711 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:38.838286 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:38.838711 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:38.838396 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:38.838711 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:38.838400 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:38.838711 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:38.838463 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:54.838443642 +0000 UTC m=+34.149061160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:38.838711 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:38.838479 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret podName:a2e19c4a-e0b3-46a7-b8de-563c6b248018 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:54.838472853 +0000 UTC m=+34.149090355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret") pod "global-pull-secret-syncer-bjk4m" (UID: "a2e19c4a-e0b3-46a7-b8de-563c6b248018") : object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:39.039918 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:39.039829 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:39.040073 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:39.040026 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 21:51:39.040073 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:39.040055 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 21:51:39.040073 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:39.040070 2556 projected.go:196] Error preparing data for projected volume kube-api-access-b4kvj for pod openshift-network-diagnostics/network-check-target-k9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:39.040231 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:39.040139 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj podName:aa94d66c-34f4-40a5-b073-f439fe7f01ed nodeName:}" failed. No retries permitted until 2026-07-09 21:51:55.040120392 +0000 UTC m=+34.350737915 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4kvj" (UniqueName: "kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj") pod "network-check-target-k9bcs" (UID: "aa94d66c-34f4-40a5-b073-f439fe7f01ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:39.269615 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:39.269560 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:39.269615 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:39.269613 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:39.269858 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:39.269560 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:39.269858 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:39.269711 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:39.269858 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:39.269803 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:39.270003 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:39.269888 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:41.271048 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:41.271008 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:41.271595 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:41.271115 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:41.271595 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:41.271124 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:41.271595 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:41.271158 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:41.271595 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:41.271213 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:41.271595 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:41.271334 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:43.270229 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.270078 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:43.270943 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.270078 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:43.270943 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:43.270308 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:43.270943 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.270115 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:43.270943 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:43.270402 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:43.270943 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:43.270484 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:43.529683 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.529409 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bntnp" event={"ID":"fd78623e-cb49-4f50-ae98-3eb77e2c762e","Type":"ContainerStarted","Data":"cfede399d0115b3d14d3afe61c918a77b8c82bbf654dfb9950bafc3f7c602e59"} Jul 09 21:51:43.530859 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.530838 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-58pcd" event={"ID":"19ccaed5-cce9-46ff-9865-09829405516f","Type":"ContainerStarted","Data":"5a805bbb2328256b9fa2edf1fefcced12f09f266d4128611d920e9f136f58074"} Jul 09 21:51:43.532195 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.532174 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dp58g" event={"ID":"95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b","Type":"ContainerStarted","Data":"b821086cd439a77901a39f92314c6b0f4b55f8d5c85c68584b463bb75833922c"} Jul 09 21:51:43.534985 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.534960 2556 generic.go:356] "Generic (PLEG): container finished" podID="82a91008-923a-4f71-9420-b743b9a391e9" containerID="54b0752265c9e342b791cb171d8410050fce05caae318338f134f807723ea306" exitCode=0 Jul 09 21:51:43.535082 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.535037 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" event={"ID":"82a91008-923a-4f71-9420-b743b9a391e9","Type":"ContainerDied","Data":"54b0752265c9e342b791cb171d8410050fce05caae318338f134f807723ea306"} Jul 09 21:51:43.536569 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.536539 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9znf9" event={"ID":"a2ca5280-d3ac-4cec-82ed-67dcdfff85f8","Type":"ContainerStarted","Data":"6b6429f74efabda59b7fb0f33eb99240d7811f0a958110dcf43d1bbd25f6b619"} Jul 09 21:51:43.538080 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.538062 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" event={"ID":"0d9060a4-7039-4b8a-a3af-118f7aa90010","Type":"ContainerStarted","Data":"c04feb7bad98e1eea06ba91985583bd4d1b36a00c4a8a4eec141906e29dfb702"} Jul 09 21:51:43.540693 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.540673 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"921c2c036fb67b6600931cab3821b2fd4a1183eaaf7d883e104b2d49be44dc8b"} Jul 09 21:51:43.540788 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.540732 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"525eaa648ce02a6ea30994570e98cf433144bbb5479e015fd747a3b92d78814f"} Jul 09 21:51:43.540788 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.540774 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"80e24587ef669fa9b6f91c131ca9b5ff1aded251f3c1d67578c74fb00b40fd82"} Jul 09 21:51:43.540905 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.540817 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"4699340db8f2923c1cbc24595bf9d3b4040dbcc511c794ebe67ec480e8f3572a"} Jul 09 21:51:43.540905 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.540855 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"cbeb385690080611bcc80b89bf82651d775247484e6000c7afc894534a7f1625"} Jul 09 21:51:43.542091 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.542072 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fvpk9" event={"ID":"1c4578ac-8173-4dbc-8332-250b5a40fd7a","Type":"ContainerStarted","Data":"7ba6529dafd53446714acd4d4502a08da75762d8931b556c0c30c051c109cce8"} Jul 09 21:51:43.547605 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.545486 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bntnp" podStartSLOduration=3.970983846 podStartE2EDuration="22.545469223s" totalImagesPullingTime="18.574485377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:43.544745314 +0000 UTC m=+22.855362838" watchObservedRunningTime="2026-07-09 21:51:43.545469223 +0000 UTC m=+22.856086747" Jul 09 21:51:43.562422 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.562385 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fvpk9" podStartSLOduration=8.410536464 podStartE2EDuration="22.562372913s" totalImagesPullingTime="14.151836449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:43.562268867 +0000 UTC m=+22.872886390" watchObservedRunningTime="2026-07-09 21:51:43.562372913 +0000 UTC m=+22.872990458" Jul 09 21:51:43.577154 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.577115 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-58pcd" podStartSLOduration=4.055253498 podStartE2EDuration="22.577101012s" totalImagesPullingTime="18.521847514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:43.576510954 +0000 UTC m=+22.887128478" watchObservedRunningTime="2026-07-09 21:51:43.577101012 +0000 UTC m=+22.887718532" Jul 09 21:51:43.612124 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.612082 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9znf9" podStartSLOduration=4.08115381 podStartE2EDuration="22.612067475s" totalImagesPullingTime="18.530913665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:43.611479697 +0000 UTC m=+22.922097221" watchObservedRunningTime="2026-07-09 21:51:43.612067475 +0000 UTC m=+22.922684998" Jul 09 21:51:43.625291 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:43.625247 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dp58g" podStartSLOduration=4.106494376 podStartE2EDuration="22.625234378s" totalImagesPullingTime="18.518740002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:43.624985035 +0000 UTC m=+22.935602558" watchObservedRunningTime="2026-07-09 21:51:43.625234378 +0000 UTC m=+22.935851892" Jul 09 21:51:44.372165 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:44.372139 2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 21:51:44.547258 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:44.547174 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" event={"ID":"0d9060a4-7039-4b8a-a3af-118f7aa90010","Type":"ContainerStarted","Data":"64b7a451e13b11fd79282d9e3e6338ea2a98e050f5835f781a579138ec21f120"} Jul 09 21:51:44.550636 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:44.550597 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"958054c8bc0e00279ad483a42099161eef8d930f7b5ecea87e3ded32d51939e7"} Jul 09 21:51:44.552337 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:44.552303 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kghj8" event={"ID":"8b9736d4-bfff-4aa5-a935-f5691adee10f","Type":"ContainerStarted","Data":"a1a053da09f200c05ed34a4be74bec0c4cb33ce6526afc936a683a967333991a"} Jul 09 21:51:44.566698 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:44.566656 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kghj8" podStartSLOduration=5.053269274 podStartE2EDuration="23.566641533s" totalImagesPullingTime="18.513372259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:44.566127979 +0000 UTC m=+23.876745502" watchObservedRunningTime="2026-07-09 21:51:44.566641533 +0000 UTC m=+23.877259057" Jul 09 21:51:45.164933 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:45.164813 2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T21:51:44.37216122Z","UUID":"35698c21-e41c-444d-b991-3ba956ac64d0","Handler":null,"Name":"","Endpoint":""} Jul 09 21:51:45.166281 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:45.166259 2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 21:51:45.166396 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:45.166295 2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 21:51:45.269782 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:45.269745 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:45.270038 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:45.269745 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:45.270038 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:45.269881 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:45.270038 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:45.269752 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:45.270038 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:45.269956 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:45.270259 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:45.270040 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:45.588785 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:45.588749 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:45.589312 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:45.589290 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:46.560362 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:46.560329 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" event={"ID":"0d9060a4-7039-4b8a-a3af-118f7aa90010","Type":"ContainerStarted","Data":"3d1a8e8e54f59a2f4ca97403cff94042786f7b5a352a572c54a70da0f8b6da67"} Jul 09 21:51:46.563998 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:46.563965 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"3bb11f56d512ce3ce805ade4700eb02f6c002497943961563ed1cc51d95abea1"} Jul 09 21:51:46.564167 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:46.564054 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:46.564719 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:46.564700 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fvpk9" Jul 09 21:51:46.579045 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:46.579000 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lsjk" podStartSLOduration=4.761926858 podStartE2EDuration="25.578987111s" totalImagesPullingTime="20.817060253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:46.578722739 +0000 UTC m=+25.889340269" watchObservedRunningTime="2026-07-09 21:51:46.578987111 +0000 UTC m=+25.889604637" Jul 09 21:51:47.269361 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:47.269324 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:47.269876 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:47.269327 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:47.269876 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:47.269467 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:47.269876 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:47.269337 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:47.269876 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:47.269502 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:47.269876 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:47.269638 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:48.571567 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:48.571346 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" event={"ID":"bb908ed8-23f3-496e-a282-218bd4208b2d","Type":"ContainerStarted","Data":"5d4bea215ec397f6d7a3cd9f05ee9726bc3c6f526fc1913b031b61d6bc131fd3"} Jul 09 21:51:48.597701 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:48.597642 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" podStartSLOduration=9.01345385 podStartE2EDuration="27.597624147s" totalImagesPullingTime="18.584170297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:51:48.596993309 +0000 UTC m=+27.907610832" watchObservedRunningTime="2026-07-09 21:51:48.597624147 +0000 UTC m=+27.908241671" Jul 09 21:51:49.271190 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.271157 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:49.271380 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.271285 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:49.271451 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:49.271403 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:49.271451 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:49.271287 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:49.271451 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.271168 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:49.271605 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:49.271492 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:49.573780 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.573713 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 21:51:49.574246 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.574218 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:49.574339 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.574320 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:49.589985 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.589806 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:49.590105 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.590062 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:49.930109 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.930078 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjk4m"] Jul 09 21:51:49.930277 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.930183 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:49.930338 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:49.930294 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:49.934806 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.934781 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k9bcs"] Jul 09 21:51:49.934932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.934863 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:49.935003 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:49.934940 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:49.935667 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.935642 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5jcvb"] Jul 09 21:51:49.935902 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:49.935724 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:49.935902 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:49.935839 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:50.577250 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:50.577214 2556 generic.go:356] "Generic (PLEG): container finished" podID="82a91008-923a-4f71-9420-b743b9a391e9" containerID="ba2ddec654ab1f5597f8a7ff7bc9d60f4c5534c4df7f33e1ddca41f370aa0573" exitCode=0 Jul 09 21:51:50.577664 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:50.577293 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" event={"ID":"82a91008-923a-4f71-9420-b743b9a391e9","Type":"ContainerDied","Data":"ba2ddec654ab1f5597f8a7ff7bc9d60f4c5534c4df7f33e1ddca41f370aa0573"} Jul 09 21:51:50.577664 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:50.577390 2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 21:51:51.174651 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:51.174614 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:51:51.270780 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:51.270749 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:51.270906 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:51.270858 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:51.270946 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:51.270907 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:51.271005 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:51.270987 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:52.269886 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:52.269853 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:52.270302 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:52.269980 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:52.584656 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:52.584603 2556 generic.go:356] "Generic (PLEG): container finished" podID="82a91008-923a-4f71-9420-b743b9a391e9" containerID="574713920d3a1ce1a005e8388a11697971cc6f9d0d050b232f298c19956803f9" exitCode=0 Jul 09 21:51:52.586389 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:52.586248 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" event={"ID":"82a91008-923a-4f71-9420-b743b9a391e9","Type":"ContainerDied","Data":"574713920d3a1ce1a005e8388a11697971cc6f9d0d050b232f298c19956803f9"} Jul 09 21:51:52.606054 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:52.605889 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" podUID="bb908ed8-23f3-496e-a282-218bd4208b2d" containerName="ovnkube-controller" probeResult="failure" output="" Jul 09 21:51:53.269825 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:53.269793 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:53.269975 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:53.269837 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:53.269975 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:53.269907 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:53.270244 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:53.270022 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:54.269634 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:54.269569 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:54.269842 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:54.269721 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjk4m" podUID="a2e19c4a-e0b3-46a7-b8de-563c6b248018" Jul 09 21:51:54.591219 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:54.591188 2556 generic.go:356] "Generic (PLEG): container finished" podID="82a91008-923a-4f71-9420-b743b9a391e9" containerID="4d7eedec8e04f5c9d7174d637e7c9420377d8c6b19cc3930aa5d8830fc5441c3" exitCode=0 Jul 09 21:51:54.591717 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:54.591224 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" event={"ID":"82a91008-923a-4f71-9420-b743b9a391e9","Type":"ContainerDied","Data":"4d7eedec8e04f5c9d7174d637e7c9420377d8c6b19cc3930aa5d8830fc5441c3"} Jul 09 21:51:54.869594 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:54.869482 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:54.869594 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:54.869539 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:54.869802 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:54.869641 2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:54.869802 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:54.869679 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:54.869802 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:54.869704 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret podName:a2e19c4a-e0b3-46a7-b8de-563c6b248018 nodeName:}" failed. No retries permitted until 2026-07-09 21:52:26.869688009 +0000 UTC m=+66.180305510 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret") pod "global-pull-secret-syncer-bjk4m" (UID: "a2e19c4a-e0b3-46a7-b8de-563c6b248018") : object "kube-system"/"original-pull-secret" not registered Jul 09 21:51:54.869802 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:54.869727 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:52:26.869711541 +0000 UTC m=+66.180329055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 21:51:55.070896 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.070856 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:55.071068 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:55.071014 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 21:51:55.071068 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:55.071030 2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 21:51:55.071068 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:55.071040 2556 projected.go:196] Error preparing data for projected volume kube-api-access-b4kvj for pod openshift-network-diagnostics/network-check-target-k9bcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:55.071162 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:55.071092 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj podName:aa94d66c-34f4-40a5-b073-f439fe7f01ed nodeName:}" failed. No retries permitted until 2026-07-09 21:52:27.071075435 +0000 UTC m=+66.381692940 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4kvj" (UniqueName: "kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj") pod "network-check-target-k9bcs" (UID: "aa94d66c-34f4-40a5-b073-f439fe7f01ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 21:51:55.270310 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.270231 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:55.270443 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:55.270359 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k9bcs" podUID="aa94d66c-34f4-40a5-b073-f439fe7f01ed" Jul 09 21:51:55.270443 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.270408 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:55.270530 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:55.270503 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:51:55.949302 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.949269 2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-160.ec2.internal" event="NodeReady" Jul 09 21:51:55.949768 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.949429 2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 21:51:55.994887 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.994835 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b4sg8"] Jul 09 21:51:55.997517 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.997493 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:55.997671 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.997604 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8n2cw"] Jul 09 21:51:55.999133 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:55.999117 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8n2cw" Jul 09 21:51:56.000509 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.000493 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 21:51:56.000609 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.000492 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 21:51:56.000655 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.000623 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4k4f4\"" Jul 09 21:51:56.001409 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.001377 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 21:51:56.001500 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.001444 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vkm9l\"" Jul 09 21:51:56.001500 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.001388 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 21:51:56.001500 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.001453 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 21:51:56.006894 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.006876 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b4sg8"] Jul 09 21:51:56.010369 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.010288 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8n2cw"] Jul 09 21:51:56.179869 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.179836 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.179869 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.179866 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4488e1dc-e74a-48a7-a208-e6371f9d4375-tmp-dir\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.180057 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.179884 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvjr\" (UniqueName: \"kubernetes.io/projected/4488e1dc-e74a-48a7-a208-e6371f9d4375-kube-api-access-phvjr\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.180057 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.179989 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwrc\" (UniqueName: \"kubernetes.io/projected/115dcab9-3d51-43fa-ab57-c00a61331fc3-kube-api-access-2dwrc\") pod \"ingress-canary-8n2cw\" (UID: \"115dcab9-3d51-43fa-ab57-c00a61331fc3\") " pod="openshift-ingress-canary/ingress-canary-8n2cw" Jul 09 21:51:56.180057 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.180035 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/115dcab9-3d51-43fa-ab57-c00a61331fc3-cert\") pod \"ingress-canary-8n2cw\" (UID: \"115dcab9-3d51-43fa-ab57-c00a61331fc3\") " pod="openshift-ingress-canary/ingress-canary-8n2cw" Jul 09 21:51:56.180160 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.180093 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4488e1dc-e74a-48a7-a208-e6371f9d4375-config-volume\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.270184 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.270112 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:51:56.272866 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.272850 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 21:51:56.280780 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.280758 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/115dcab9-3d51-43fa-ab57-c00a61331fc3-cert\") pod \"ingress-canary-8n2cw\" (UID: \"115dcab9-3d51-43fa-ab57-c00a61331fc3\") " pod="openshift-ingress-canary/ingress-canary-8n2cw" Jul 09 21:51:56.280885 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.280823 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4488e1dc-e74a-48a7-a208-e6371f9d4375-config-volume\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.280885 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.280873 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.280994 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.280898 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4488e1dc-e74a-48a7-a208-e6371f9d4375-tmp-dir\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.280994 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.280922 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvjr\" (UniqueName: \"kubernetes.io/projected/4488e1dc-e74a-48a7-a208-e6371f9d4375-kube-api-access-phvjr\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.280994 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.280965 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwrc\" (UniqueName: \"kubernetes.io/projected/115dcab9-3d51-43fa-ab57-c00a61331fc3-kube-api-access-2dwrc\") pod \"ingress-canary-8n2cw\" (UID: \"115dcab9-3d51-43fa-ab57-c00a61331fc3\") " pod="openshift-ingress-canary/ingress-canary-8n2cw" Jul 09 21:51:56.281141 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:56.280991 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:51:56.281141 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:56.281060 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:56.781038604 +0000 UTC m=+36.091656127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:51:56.281224 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.281186 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4488e1dc-e74a-48a7-a208-e6371f9d4375-tmp-dir\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.281333 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.281317 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4488e1dc-e74a-48a7-a208-e6371f9d4375-config-volume\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.284757 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.284741 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/115dcab9-3d51-43fa-ab57-c00a61331fc3-cert\") pod \"ingress-canary-8n2cw\" (UID: \"115dcab9-3d51-43fa-ab57-c00a61331fc3\") " pod="openshift-ingress-canary/ingress-canary-8n2cw" Jul 09 21:51:56.288691 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.288658 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvjr\" (UniqueName: \"kubernetes.io/projected/4488e1dc-e74a-48a7-a208-e6371f9d4375-kube-api-access-phvjr\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.288968 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.288946 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwrc\" (UniqueName: \"kubernetes.io/projected/115dcab9-3d51-43fa-ab57-c00a61331fc3-kube-api-access-2dwrc\") pod \"ingress-canary-8n2cw\" (UID: \"115dcab9-3d51-43fa-ab57-c00a61331fc3\") " pod="openshift-ingress-canary/ingress-canary-8n2cw" Jul 09 21:51:56.313760 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.313718 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8n2cw" Jul 09 21:51:56.468218 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.468186 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8n2cw"] Jul 09 21:51:56.473887 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:51:56.473859 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115dcab9_3d51_43fa_ab57_c00a61331fc3.slice/crio-cd6651eeff237b2ee8c243f4745368bdaddce999f59ccf224bb46b6cec944df9 WatchSource:0}: Error finding container cd6651eeff237b2ee8c243f4745368bdaddce999f59ccf224bb46b6cec944df9: Status 404 returned error can't find the container with id cd6651eeff237b2ee8c243f4745368bdaddce999f59ccf224bb46b6cec944df9 Jul 09 21:51:56.597238 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.597206 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8n2cw" event={"ID":"115dcab9-3d51-43fa-ab57-c00a61331fc3","Type":"ContainerStarted","Data":"cd6651eeff237b2ee8c243f4745368bdaddce999f59ccf224bb46b6cec944df9"} Jul 09 21:51:56.787178 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:56.787141 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:56.787342 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:56.787320 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:51:56.787409 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:56.787401 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:57.787378134 +0000 UTC m=+37.097995651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:51:57.270306 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:57.270241 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:51:57.270306 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:57.270262 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:51:57.273122 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:57.273097 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r7ms8\"" Jul 09 21:51:57.274016 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:57.273997 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 21:51:57.274145 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:57.274101 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 21:51:57.274145 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:57.274118 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdkql\"" Jul 09 21:51:57.274272 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:57.274174 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 21:51:57.794866 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:57.794642 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:57.795059 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:57.794802 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:51:57.795059 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:57.794968 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:51:59.794949475 +0000 UTC m=+39.105566983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:51:59.811081 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:51:59.810975 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:51:59.811564 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:59.811152 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:51:59.811564 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:51:59.811238 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:52:03.8112154 +0000 UTC m=+43.121832924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:52:01.611297 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:01.611258 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8n2cw" event={"ID":"115dcab9-3d51-43fa-ab57-c00a61331fc3","Type":"ContainerStarted","Data":"4c40435d896984fa32cbc50cb7e2de9a0cfa2ee5e9755b143d47d74bb21925fa"} Jul 09 21:52:01.614106 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:01.614085 2556 generic.go:356] "Generic (PLEG): container finished" podID="82a91008-923a-4f71-9420-b743b9a391e9" containerID="6609ee5f990a4318375bdcb8f5eca28bfaad15736ebf6d9f3d2424dd617a2353" exitCode=0 Jul 09 21:52:01.614183 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:01.614117 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" event={"ID":"82a91008-923a-4f71-9420-b743b9a391e9","Type":"ContainerDied","Data":"6609ee5f990a4318375bdcb8f5eca28bfaad15736ebf6d9f3d2424dd617a2353"} Jul 09 21:52:01.626222 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:01.626181 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8n2cw" podStartSLOduration=2.4846122409999998 podStartE2EDuration="6.626168041s" totalImagesPullingTime="4.1415558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:52:01.625297234 +0000 UTC m=+40.935914759" watchObservedRunningTime="2026-07-09 21:52:01.626168041 +0000 UTC m=+40.936785563" Jul 09 21:52:02.618977 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:02.618942 2556 generic.go:356] "Generic (PLEG): container finished" podID="82a91008-923a-4f71-9420-b743b9a391e9" containerID="f7429b70bf447dcdc29bdec465c63938fb6de0e0f346410851dbc23c33955642" exitCode=0 Jul 09 21:52:02.619362 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:02.619030 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" event={"ID":"82a91008-923a-4f71-9420-b743b9a391e9","Type":"ContainerDied","Data":"f7429b70bf447dcdc29bdec465c63938fb6de0e0f346410851dbc23c33955642"} Jul 09 21:52:03.624505 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:03.624465 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" event={"ID":"82a91008-923a-4f71-9420-b743b9a391e9","Type":"ContainerStarted","Data":"8aeb8c2d938ae5b3c13e3c7e54c69a032e0143afb9d02e2a8d2c613023e87c01"} Jul 09 21:52:03.647289 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:03.647249 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hzrbj" podStartSLOduration=9.661102057 podStartE2EDuration="42.647235056s" totalImagesPullingTime="32.986132999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:52:03.645837733 +0000 UTC m=+42.956455256" watchObservedRunningTime="2026-07-09 21:52:03.647235056 +0000 UTC m=+42.957852576" Jul 09 21:52:03.839364 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:03.839329 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:52:03.839530 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:03.839470 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:52:03.839594 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:03.839532 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:52:11.839515512 +0000 UTC m=+51.150133014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:52:11.899006 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:11.898966 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:52:11.899408 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:11.899125 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:52:11.899408 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:11.899203 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:52:27.899185932 +0000 UTC m=+67.209803434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:52:22.595460 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:22.595425 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dmq54" Jul 09 21:52:26.902373 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:26.902321 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:52:26.902816 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:26.902401 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:52:26.905057 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:26.905037 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 21:52:26.905171 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:26.905157 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 21:52:26.913247 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:26.913230 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 21:52:26.913331 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:26.913307 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:53:30.913285657 +0000 UTC m=+130.223903158 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : secret "metrics-daemon-secret" not found Jul 09 21:52:26.916366 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:26.916349 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a2e19c4a-e0b3-46a7-b8de-563c6b248018-original-pull-secret\") pod \"global-pull-secret-syncer-bjk4m\" (UID: \"a2e19c4a-e0b3-46a7-b8de-563c6b248018\") " pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:52:27.103628 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.103561 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:52:27.106445 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.106429 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 21:52:27.116642 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.116626 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 21:52:27.127783 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.127766 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4kvj\" (UniqueName: \"kubernetes.io/projected/aa94d66c-34f4-40a5-b073-f439fe7f01ed-kube-api-access-b4kvj\") pod \"network-check-target-k9bcs\" (UID: \"aa94d66c-34f4-40a5-b073-f439fe7f01ed\") " pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:52:27.178826 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.178744 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjk4m" Jul 09 21:52:27.296339 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.296310 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjk4m"] Jul 09 21:52:27.297901 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.297886 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r7ms8\"" Jul 09 21:52:27.299445 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:52:27.299419 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e19c4a_e0b3_46a7_b8de_563c6b248018.slice/crio-01a054eb57947e061f5fea9607fd7d3c292d13afb9c150e979380b614991198a WatchSource:0}: Error finding container 01a054eb57947e061f5fea9607fd7d3c292d13afb9c150e979380b614991198a: Status 404 returned error can't find the container with id 01a054eb57947e061f5fea9607fd7d3c292d13afb9c150e979380b614991198a Jul 09 21:52:27.306547 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.306528 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:52:27.423690 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.423659 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k9bcs"] Jul 09 21:52:27.426434 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:52:27.426398 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa94d66c_34f4_40a5_b073_f439fe7f01ed.slice/crio-8450b42f0c2604eca32418514efc96ccdf338466e3a63d04d55ebe082a60f061 WatchSource:0}: Error finding container 8450b42f0c2604eca32418514efc96ccdf338466e3a63d04d55ebe082a60f061: Status 404 returned error can't find the container with id 8450b42f0c2604eca32418514efc96ccdf338466e3a63d04d55ebe082a60f061 Jul 09 21:52:27.681716 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.681531 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k9bcs" event={"ID":"aa94d66c-34f4-40a5-b073-f439fe7f01ed","Type":"ContainerStarted","Data":"8450b42f0c2604eca32418514efc96ccdf338466e3a63d04d55ebe082a60f061"} Jul 09 21:52:27.682674 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.682642 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjk4m" event={"ID":"a2e19c4a-e0b3-46a7-b8de-563c6b248018","Type":"ContainerStarted","Data":"01a054eb57947e061f5fea9607fd7d3c292d13afb9c150e979380b614991198a"} Jul 09 21:52:27.909784 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:27.909741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:52:27.910235 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:27.909909 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:52:27.910235 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:27.909968 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:52:59.909949244 +0000 UTC m=+99.220566751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:52:32.695957 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:32.695922 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k9bcs" event={"ID":"aa94d66c-34f4-40a5-b073-f439fe7f01ed","Type":"ContainerStarted","Data":"928e3fa7ee7f1dcd15b65e774c785eb249728a53881ce27a63d6c84df45e2633"} Jul 09 21:52:32.696454 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:32.696034 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:52:32.697348 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:32.697325 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjk4m" event={"ID":"a2e19c4a-e0b3-46a7-b8de-563c6b248018","Type":"ContainerStarted","Data":"82f5d6a4ef8a3ee4b9d7268f2dcf82bc14cdd6ea262ce489515a8fb764f4bc7d"} Jul 09 21:52:32.711629 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:32.711549 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k9bcs" podStartSLOduration=66.912896585 podStartE2EDuration="1m11.711536101s" totalImagesPullingTime="4.798639516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:52:32.711439245 +0000 UTC m=+72.022056769" watchObservedRunningTime="2026-07-09 21:52:32.711536101 +0000 UTC m=+72.022153623" Jul 09 21:52:32.726149 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:32.726097 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bjk4m" podStartSLOduration=66.797226807 podStartE2EDuration="1m11.726078568s" totalImagesPullingTime="4.928851761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:52:32.724713769 +0000 UTC m=+72.035331295" watchObservedRunningTime="2026-07-09 21:52:32.726078568 +0000 UTC m=+72.036696093" Jul 09 21:52:59.923866 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:52:59.923816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:52:59.924327 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:59.923946 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:52:59.924327 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:52:59.924003 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:54:03.923989484 +0000 UTC m=+163.234606988 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:53:03.701538 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:53:03.701507 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k9bcs" Jul 09 21:53:30.934248 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:53:30.934207 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:53:30.934851 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:53:30.934324 2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 21:53:30.934851 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:53:30.934378 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs podName:ceeea23b-5ed9-45c1-97c0-c223a39f03b9 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:32.934363249 +0000 UTC m=+252.244980754 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs") pod "network-metrics-daemon-5jcvb" (UID: "ceeea23b-5ed9-45c1-97c0-c223a39f03b9") : secret "metrics-daemon-secret" not found Jul 09 21:53:59.008417 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:53:59.008372 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b4sg8" podUID="4488e1dc-e74a-48a7-a208-e6371f9d4375" Jul 09 21:53:59.894383 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:53:59.894357 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4sg8" Jul 09 21:54:00.284208 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:54:00.284124 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5jcvb" podUID="ceeea23b-5ed9-45c1-97c0-c223a39f03b9" Jul 09 21:54:03.955463 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:54:03.955406 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:54:03.955910 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:54:03.955558 2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 21:54:03.955910 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:54:03.955638 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls podName:4488e1dc-e74a-48a7-a208-e6371f9d4375 nodeName:}" failed. No retries permitted until 2026-07-09 21:56:05.955622301 +0000 UTC m=+285.266239803 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls") pod "dns-default-b4sg8" (UID: "4488e1dc-e74a-48a7-a208-e6371f9d4375") : secret "dns-default-metrics-tls" not found Jul 09 21:54:11.270593 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:54:11.270546 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:55:14.664447 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.664417 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk"] Jul 09 21:55:14.666208 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.666184 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-8q4rh"] Jul 09 21:55:14.666334 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.666318 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.667615 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.667599 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26"] Jul 09 21:55:14.667747 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.667728 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.668747 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.668727 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 21:55:14.668851 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.668805 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 21:55:14.668955 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.668937 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 21:55:14.669076 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.669010 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 21:55:14.669076 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.669015 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-n5hfz\"" Jul 09 21:55:14.669206 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.669191 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc"] Jul 09 21:55:14.669265 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.669252 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:14.669800 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.669784 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 21:55:14.669800 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.669794 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 21:55:14.669931 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.669842 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 21:55:14.670132 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.670120 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 21:55:14.670828 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.670806 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc" Jul 09 21:55:14.671073 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.671057 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-kqf8n\"" Jul 09 21:55:14.671508 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.671480 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9hrd7\"" Jul 09 21:55:14.671645 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.671515 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 21:55:14.671645 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.671528 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 21:55:14.672174 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.672155 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 21:55:14.675879 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.675815 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 21:55:14.675979 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.675936 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 21:55:14.675979 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.675967 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 21:55:14.676128 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.676028 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2srvd\"" Jul 09 21:55:14.687418 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.687394 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-8q4rh"] Jul 09 21:55:14.688132 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.688115 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc"] Jul 09 21:55:14.690154 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.690124 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk"] Jul 09 21:55:14.700783 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.700760 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26"] Jul 09 21:55:14.716085 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.716054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkgx\" (UniqueName: \"kubernetes.io/projected/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-kube-api-access-8nkgx\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.716215 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.716121 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.716215 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.716167 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.763264 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.763227 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-xfqh4"] Jul 09 21:55:14.765329 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.765312 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b5486fbd7-n8ft6"] Jul 09 21:55:14.765444 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.765426 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.766904 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.766883 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:14.767766 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.767747 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 21:55:14.768088 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.768072 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 21:55:14.768168 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.768091 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 21:55:14.768227 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.768199 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 21:55:14.768324 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.768310 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r77kl\"" Jul 09 21:55:14.769208 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.769190 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 21:55:14.769459 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.769443 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 21:55:14.770410 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.770389 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k7jf4\"" Jul 09 21:55:14.770410 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.770402 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 21:55:14.770609 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.770426 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 21:55:14.770609 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.770483 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 21:55:14.770609 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.770545 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 21:55:14.775447 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.775427 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 21:55:14.780907 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.780885 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-xfqh4"] Jul 09 21:55:14.781435 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.781417 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b5486fbd7-n8ft6"] Jul 09 21:55:14.817197 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817160 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.817411 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817387 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04d3e78f-c77a-4677-8b22-643f3e657964-serving-cert\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.817474 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817432 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f715282-1ab7-427e-973a-550762cade1e-trusted-ca\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.817474 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817465 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj475\" (UniqueName: \"kubernetes.io/projected/4f715282-1ab7-427e-973a-550762cade1e-kube-api-access-mj475\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.817567 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817495 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d3e78f-c77a-4677-8b22-643f3e657964-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.817567 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kps\" (UniqueName: \"kubernetes.io/projected/3563e31a-0ecb-4d93-8cc0-c393645a63e2-kube-api-access-n8kps\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:14.817712 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817640 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d3e78f-c77a-4677-8b22-643f3e657964-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.817712 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817665 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86w5\" (UniqueName: \"kubernetes.io/projected/04d3e78f-c77a-4677-8b22-643f3e657964-kube-api-access-r86w5\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.817802 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817730 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/04d3e78f-c77a-4677-8b22-643f3e657964-snapshots\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.817802 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817761 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkgx\" (UniqueName: \"kubernetes.io/projected/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-kube-api-access-8nkgx\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.817802 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817793 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04d3e78f-c77a-4677-8b22-643f3e657964-tmp\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.817932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817823 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:14.817932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817853 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f715282-1ab7-427e-973a-550762cade1e-serving-cert\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.817932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817871 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.817932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817888 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mds22\" (UniqueName: \"kubernetes.io/projected/3760cd42-b37d-490a-85ef-4c446d344a5e-kube-api-access-mds22\") pod \"volume-data-source-validator-5d456fdbcd-cl9rc\" (UID: \"3760cd42-b37d-490a-85ef-4c446d344a5e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc" Jul 09 21:55:14.818103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.817936 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f715282-1ab7-427e-973a-550762cade1e-config\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.818103 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.818074 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.820970 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.820945 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.824518 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.824499 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkgx\" (UniqueName: \"kubernetes.io/projected/520a37a8-e07d-41c7-99c5-51bcbdd6ef23-kube-api-access-8nkgx\") pod \"kube-storage-version-migrator-operator-846bf668f7-6kzqk\" (UID: \"520a37a8-e07d-41c7-99c5-51bcbdd6ef23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.919533 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/04d3e78f-c77a-4677-8b22-643f3e657964-snapshots\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.919533 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f715282-1ab7-427e-973a-550762cade1e-serving-cert\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.919533 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919515 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-stats-auth\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:14.919533 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919533 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d3e78f-c77a-4677-8b22-643f3e657964-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.919835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919551 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kps\" (UniqueName: \"kubernetes.io/projected/3563e31a-0ecb-4d93-8cc0-c393645a63e2-kube-api-access-n8kps\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:14.919835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919568 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:14.919835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919624 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04d3e78f-c77a-4677-8b22-643f3e657964-tmp\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.919835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919653 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d3e78f-c77a-4677-8b22-643f3e657964-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.919835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919753 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:14.919835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919792 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04d3e78f-c77a-4677-8b22-643f3e657964-serving-cert\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.919835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919815 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f715282-1ab7-427e-973a-550762cade1e-trusted-ca\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.920154 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.919844 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-default-certificate\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:14.920154 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:14.920002 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 21:55:14.920154 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:14.920079 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls podName:3563e31a-0ecb-4d93-8cc0-c393645a63e2 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:15.420057649 +0000 UTC m=+234.730675155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b2m26" (UID: "3563e31a-0ecb-4d93-8cc0-c393645a63e2") : secret "samples-operator-tls" not found Jul 09 21:55:14.920303 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920207 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/04d3e78f-c77a-4677-8b22-643f3e657964-snapshots\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.920355 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920323 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d3e78f-c77a-4677-8b22-643f3e657964-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.920398 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920374 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86w5\" (UniqueName: \"kubernetes.io/projected/04d3e78f-c77a-4677-8b22-643f3e657964-kube-api-access-r86w5\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.920435 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920414 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:14.920488 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920442 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f715282-1ab7-427e-973a-550762cade1e-config\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.920488 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920476 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpr9\" (UniqueName: \"kubernetes.io/projected/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-kube-api-access-7vpr9\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:14.920607 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920502 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04d3e78f-c77a-4677-8b22-643f3e657964-tmp\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.920607 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920509 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mds22\" (UniqueName: \"kubernetes.io/projected/3760cd42-b37d-490a-85ef-4c446d344a5e-kube-api-access-mds22\") pod \"volume-data-source-validator-5d456fdbcd-cl9rc\" (UID: \"3760cd42-b37d-490a-85ef-4c446d344a5e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc" Jul 09 21:55:14.920607 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920529 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04d3e78f-c77a-4677-8b22-643f3e657964-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.920607 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920563 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj475\" (UniqueName: \"kubernetes.io/projected/4f715282-1ab7-427e-973a-550762cade1e-kube-api-access-mj475\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.920797 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.920775 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4f715282-1ab7-427e-973a-550762cade1e-trusted-ca\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.921215 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.921193 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f715282-1ab7-427e-973a-550762cade1e-config\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.921966 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.921949 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f715282-1ab7-427e-973a-550762cade1e-serving-cert\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.922296 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.922273 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04d3e78f-c77a-4677-8b22-643f3e657964-serving-cert\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.931127 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.931101 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86w5\" (UniqueName: \"kubernetes.io/projected/04d3e78f-c77a-4677-8b22-643f3e657964-kube-api-access-r86w5\") pod \"insights-operator-5c847dcfc4-8q4rh\" (UID: \"04d3e78f-c77a-4677-8b22-643f3e657964\") " pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.931446 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.931426 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kps\" (UniqueName: \"kubernetes.io/projected/3563e31a-0ecb-4d93-8cc0-c393645a63e2-kube-api-access-n8kps\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:14.931922 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.931901 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj475\" (UniqueName: \"kubernetes.io/projected/4f715282-1ab7-427e-973a-550762cade1e-kube-api-access-mj475\") pod \"console-operator-68b757fb67-xfqh4\" (UID: \"4f715282-1ab7-427e-973a-550762cade1e\") " pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:14.932137 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.932117 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mds22\" (UniqueName: \"kubernetes.io/projected/3760cd42-b37d-490a-85ef-4c446d344a5e-kube-api-access-mds22\") pod \"volume-data-source-validator-5d456fdbcd-cl9rc\" (UID: \"3760cd42-b37d-490a-85ef-4c446d344a5e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc" Jul 09 21:55:14.977019 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.976992 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" Jul 09 21:55:14.982944 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.982926 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" Jul 09 21:55:14.995657 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:14.995636 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc" Jul 09 21:55:15.021343 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.021310 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-stats-auth\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.021500 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.021357 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.021500 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.021450 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-default-certificate\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.021500 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.021486 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.021691 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.021519 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpr9\" (UniqueName: \"kubernetes.io/projected/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-kube-api-access-7vpr9\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.021691 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:15.021612 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:15.521565967 +0000 UTC m=+234.832183486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : configmap references non-existent config key: service-ca.crt Jul 09 21:55:15.021887 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:15.021858 2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 21:55:15.021978 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:15.021941 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:15.521922711 +0000 UTC m=+234.832540216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : secret "router-metrics-certs-default" not found Jul 09 21:55:15.024392 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.024343 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-default-certificate\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.025896 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.025850 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-stats-auth\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.031933 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.031886 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpr9\" (UniqueName: \"kubernetes.io/projected/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-kube-api-access-7vpr9\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.075625 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.075534 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:15.117135 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.117024 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk"] Jul 09 21:55:15.122279 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:15.122215 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520a37a8_e07d_41c7_99c5_51bcbdd6ef23.slice/crio-e0929cbe5fbf0134fd0e54c9312e3749772c0c12385b1fe27f715b24c1a10a26 WatchSource:0}: Error finding container e0929cbe5fbf0134fd0e54c9312e3749772c0c12385b1fe27f715b24c1a10a26: Status 404 returned error can't find the container with id e0929cbe5fbf0134fd0e54c9312e3749772c0c12385b1fe27f715b24c1a10a26 Jul 09 21:55:15.131364 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.131341 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-8q4rh"] Jul 09 21:55:15.135106 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:15.135071 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d3e78f_c77a_4677_8b22_643f3e657964.slice/crio-c1b72d0e2984af74f11307317bee1cc6bf3a84e76d83d8c084572489c09fe8e2 WatchSource:0}: Error finding container c1b72d0e2984af74f11307317bee1cc6bf3a84e76d83d8c084572489c09fe8e2: Status 404 returned error can't find the container with id c1b72d0e2984af74f11307317bee1cc6bf3a84e76d83d8c084572489c09fe8e2 Jul 09 21:55:15.154020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.153990 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc"] Jul 09 21:55:15.156923 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:15.156892 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3760cd42_b37d_490a_85ef_4c446d344a5e.slice/crio-39689439f354a68ea7ee3c2b7bb71e7c82833abdfd9b48c625e3b9aadbcdb969 WatchSource:0}: Error finding container 39689439f354a68ea7ee3c2b7bb71e7c82833abdfd9b48c625e3b9aadbcdb969: Status 404 returned error can't find the container with id 39689439f354a68ea7ee3c2b7bb71e7c82833abdfd9b48c625e3b9aadbcdb969 Jul 09 21:55:15.207318 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.207256 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-xfqh4"] Jul 09 21:55:15.210052 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:15.210027 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f715282_1ab7_427e_973a_550762cade1e.slice/crio-990ad612c33d460af8b6ade779c601b13f4674509418ce4b3a5428b6653392c3 WatchSource:0}: Error finding container 990ad612c33d460af8b6ade779c601b13f4674509418ce4b3a5428b6653392c3: Status 404 returned error can't find the container with id 990ad612c33d460af8b6ade779c601b13f4674509418ce4b3a5428b6653392c3 Jul 09 21:55:15.423947 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.423912 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:15.424114 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:15.424034 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 21:55:15.424114 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:15.424096 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls podName:3563e31a-0ecb-4d93-8cc0-c393645a63e2 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:16.424080409 +0000 UTC m=+235.734697913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b2m26" (UID: "3563e31a-0ecb-4d93-8cc0-c393645a63e2") : secret "samples-operator-tls" not found Jul 09 21:55:15.525233 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.525149 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.525233 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:15.525199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:15.525405 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:15.525299 2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 21:55:15.525405 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:15.525326 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:16.525313436 +0000 UTC m=+235.835930936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : configmap references non-existent config key: service-ca.crt Jul 09 21:55:15.525405 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:15.525356 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:16.525342366 +0000 UTC m=+235.835959866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : secret "router-metrics-certs-default" not found Jul 09 21:55:16.077087 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:16.076993 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" event={"ID":"4f715282-1ab7-427e-973a-550762cade1e","Type":"ContainerStarted","Data":"990ad612c33d460af8b6ade779c601b13f4674509418ce4b3a5428b6653392c3"} Jul 09 21:55:16.079511 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:16.079442 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc" event={"ID":"3760cd42-b37d-490a-85ef-4c446d344a5e","Type":"ContainerStarted","Data":"39689439f354a68ea7ee3c2b7bb71e7c82833abdfd9b48c625e3b9aadbcdb969"} Jul 09 21:55:16.082963 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:16.082359 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" event={"ID":"520a37a8-e07d-41c7-99c5-51bcbdd6ef23","Type":"ContainerStarted","Data":"e0929cbe5fbf0134fd0e54c9312e3749772c0c12385b1fe27f715b24c1a10a26"} Jul 09 21:55:16.083972 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:16.083903 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" event={"ID":"04d3e78f-c77a-4677-8b22-643f3e657964","Type":"ContainerStarted","Data":"c1b72d0e2984af74f11307317bee1cc6bf3a84e76d83d8c084572489c09fe8e2"} Jul 09 21:55:16.434084 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:16.434036 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:16.434275 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:16.434230 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 21:55:16.434630 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:16.434359 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls podName:3563e31a-0ecb-4d93-8cc0-c393645a63e2 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:18.434337116 +0000 UTC m=+237.744954634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b2m26" (UID: "3563e31a-0ecb-4d93-8cc0-c393645a63e2") : secret "samples-operator-tls" not found Jul 09 21:55:16.534847 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:16.534785 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:16.534969 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:16.534906 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:16.534969 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:16.534946 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:18.534923502 +0000 UTC m=+237.845541010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : configmap references non-existent config key: service-ca.crt Jul 09 21:55:16.535167 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:16.535004 2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 21:55:16.535167 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:16.535055 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:18.535039644 +0000 UTC m=+237.845657148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : secret "router-metrics-certs-default" not found Jul 09 21:55:17.088711 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:17.088659 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc" event={"ID":"3760cd42-b37d-490a-85ef-4c446d344a5e","Type":"ContainerStarted","Data":"ebdfe61a2a1cc9e1cc2fa36a9c2c1d9405ea59d3e3f13bde59a408ce98943c59"} Jul 09 21:55:18.453372 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:18.453336 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:18.453792 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:18.453503 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 21:55:18.453792 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:18.453619 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls podName:3563e31a-0ecb-4d93-8cc0-c393645a63e2 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:22.453565508 +0000 UTC m=+241.764183022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b2m26" (UID: "3563e31a-0ecb-4d93-8cc0-c393645a63e2") : secret "samples-operator-tls" not found Jul 09 21:55:18.554267 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:18.554163 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:18.554267 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:18.554234 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:18.554488 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:18.554310 2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 21:55:18.554488 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:18.554360 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:22.554341458 +0000 UTC m=+241.864958965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : configmap references non-existent config key: service-ca.crt Jul 09 21:55:18.554488 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:18.554378 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:22.554369516 +0000 UTC m=+241.864987017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : secret "router-metrics-certs-default" not found Jul 09 21:55:19.096961 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.096925 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" event={"ID":"520a37a8-e07d-41c7-99c5-51bcbdd6ef23","Type":"ContainerStarted","Data":"4903187f82747f3f110dd9fb84896848d25ee13612d70483146a96f8dcf10e41"} Jul 09 21:55:19.098386 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.098359 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" event={"ID":"04d3e78f-c77a-4677-8b22-643f3e657964","Type":"ContainerStarted","Data":"b5117605dc571c49d593103055be547b2ac181ed804dd3b942cd1eda0ad81fd0"} Jul 09 21:55:19.099910 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.099889 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/0.log" Jul 09 21:55:19.100032 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.099917 2556 generic.go:356] "Generic (PLEG): container finished" podID="4f715282-1ab7-427e-973a-550762cade1e" containerID="fff247e0810bc5574afd45fecd108d83bc0b97ab368804164fed0fc4e8e20301" exitCode=255 Jul 09 21:55:19.100032 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.099938 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" event={"ID":"4f715282-1ab7-427e-973a-550762cade1e","Type":"ContainerDied","Data":"fff247e0810bc5574afd45fecd108d83bc0b97ab368804164fed0fc4e8e20301"} Jul 09 21:55:19.100214 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.100197 2556 scope.go:117] "RemoveContainer" containerID="fff247e0810bc5574afd45fecd108d83bc0b97ab368804164fed0fc4e8e20301" Jul 09 21:55:19.113935 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.113894 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-cl9rc" podStartSLOduration=3.7819197239999998 podStartE2EDuration="5.113877655s" totalImagesPullingTime="1.331957931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:17.103253794 +0000 UTC m=+236.413871315" watchObservedRunningTime="2026-07-09 21:55:19.113877655 +0000 UTC m=+238.424495181" Jul 09 21:55:19.114466 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.114440 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" podStartSLOduration=1.956656441 podStartE2EDuration="5.114432815s" totalImagesPullingTime="3.157776374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:19.112731402 +0000 UTC m=+238.423348925" watchObservedRunningTime="2026-07-09 21:55:19.114432815 +0000 UTC m=+238.425050334" Jul 09 21:55:19.127776 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.127731 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" podStartSLOduration=1.986591184 podStartE2EDuration="5.12771615s" totalImagesPullingTime="3.141124966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:19.127442519 +0000 UTC m=+238.438060042" watchObservedRunningTime="2026-07-09 21:55:19.12771615 +0000 UTC m=+238.438333733" Jul 09 21:55:19.846929 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.846897 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2677l"] Jul 09 21:55:19.848873 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.848858 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:19.851541 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.851516 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 21:55:19.851799 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.851782 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 21:55:19.851799 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.851799 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-78g6p\"" Jul 09 21:55:19.861640 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.861617 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2677l"] Jul 09 21:55:19.967301 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.967265 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:19.967475 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.967331 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5894dba-53ef-4271-838c-3420e112755b-crio-socket\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:19.967475 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.967356 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsfrf\" (UniqueName: \"kubernetes.io/projected/d5894dba-53ef-4271-838c-3420e112755b-kube-api-access-gsfrf\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:19.967475 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.967398 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5894dba-53ef-4271-838c-3420e112755b-data-volume\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:19.967475 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:19.967440 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.067955 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.067917 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5894dba-53ef-4271-838c-3420e112755b-crio-socket\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.068146 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.067963 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsfrf\" (UniqueName: \"kubernetes.io/projected/d5894dba-53ef-4271-838c-3420e112755b-kube-api-access-gsfrf\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.068146 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.068034 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5894dba-53ef-4271-838c-3420e112755b-data-volume\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.068146 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.068050 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5894dba-53ef-4271-838c-3420e112755b-crio-socket\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.068146 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.068064 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.068146 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.068091 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.068146 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.068140 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 21:55:20.068390 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.068197 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 21:55:20.068390 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.068201 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:20.568183166 +0000 UTC m=+239.878800667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : configmap "kube-rbac-proxy" not found Jul 09 21:55:20.068390 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.068256 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:20.568239962 +0000 UTC m=+239.878857468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : secret "insights-runtime-extractor-tls" not found Jul 09 21:55:20.068484 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.068396 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5894dba-53ef-4271-838c-3420e112755b-data-volume\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.076134 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.076112 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsfrf\" (UniqueName: \"kubernetes.io/projected/d5894dba-53ef-4271-838c-3420e112755b-kube-api-access-gsfrf\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.104597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.104530 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 21:55:20.104912 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.104895 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/0.log" Jul 09 21:55:20.104992 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.104933 2556 generic.go:356] "Generic (PLEG): container finished" podID="4f715282-1ab7-427e-973a-550762cade1e" containerID="18ad990c4e3facc040f9679dfa390cf156268bf16c1cac3bfa5ee54335f1cc8d" exitCode=255 Jul 09 21:55:20.105045 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.105011 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" event={"ID":"4f715282-1ab7-427e-973a-550762cade1e","Type":"ContainerDied","Data":"18ad990c4e3facc040f9679dfa390cf156268bf16c1cac3bfa5ee54335f1cc8d"} Jul 09 21:55:20.105093 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.105060 2556 scope.go:117] "RemoveContainer" containerID="fff247e0810bc5574afd45fecd108d83bc0b97ab368804164fed0fc4e8e20301" Jul 09 21:55:20.105200 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.105185 2556 scope.go:117] "RemoveContainer" containerID="18ad990c4e3facc040f9679dfa390cf156268bf16c1cac3bfa5ee54335f1cc8d" Jul 09 21:55:20.105381 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.105358 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-xfqh4_openshift-console-operator(4f715282-1ab7-427e-973a-550762cade1e)\"" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" podUID="4f715282-1ab7-427e-973a-550762cade1e" Jul 09 21:55:20.572016 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.571925 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.572163 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:20.572035 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:20.572163 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.572095 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 21:55:20.572163 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.572098 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 21:55:20.572163 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.572145 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:21.572132971 +0000 UTC m=+240.882750472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : configmap "kube-rbac-proxy" not found Jul 09 21:55:20.572163 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:20.572158 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:21.572152455 +0000 UTC m=+240.882769956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : secret "insights-runtime-extractor-tls" not found Jul 09 21:55:21.109066 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:21.109040 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 21:55:21.109418 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:21.109311 2556 scope.go:117] "RemoveContainer" containerID="18ad990c4e3facc040f9679dfa390cf156268bf16c1cac3bfa5ee54335f1cc8d" Jul 09 21:55:21.109459 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:21.109439 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-xfqh4_openshift-console-operator(4f715282-1ab7-427e-973a-550762cade1e)\"" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" podUID="4f715282-1ab7-427e-973a-550762cade1e" Jul 09 21:55:21.580286 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:21.580184 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:21.580286 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:21.580231 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:21.580504 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:21.580329 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 21:55:21.580504 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:21.580370 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 21:55:21.580504 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:21.580394 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:23.580378211 +0000 UTC m=+242.890995712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : configmap "kube-rbac-proxy" not found Jul 09 21:55:21.580504 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:21.580418 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:23.580403596 +0000 UTC m=+242.891021111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : secret "insights-runtime-extractor-tls" not found Jul 09 21:55:22.487854 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:22.487808 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:22.488246 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:22.487955 2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 21:55:22.488246 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:22.488018 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls podName:3563e31a-0ecb-4d93-8cc0-c393645a63e2 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:30.488001885 +0000 UTC m=+249.798619386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b2m26" (UID: "3563e31a-0ecb-4d93-8cc0-c393645a63e2") : secret "samples-operator-tls" not found Jul 09 21:55:22.589022 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:22.588982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:22.589190 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:22.589107 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:22.589190 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:22.589154 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:30.589134833 +0000 UTC m=+249.899752355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : configmap references non-existent config key: service-ca.crt Jul 09 21:55:22.589302 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:22.589201 2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 21:55:22.589302 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:22.589241 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs podName:139e71b1-5134-4bc2-8a29-96f5dc16c5b6 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:30.589229757 +0000 UTC m=+249.899847267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs") pod "router-default-6b5486fbd7-n8ft6" (UID: "139e71b1-5134-4bc2-8a29-96f5dc16c5b6") : secret "router-metrics-certs-default" not found Jul 09 21:55:22.631352 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:22.631324 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9znf9_a2ca5280-d3ac-4cec-82ed-67dcdfff85f8/dns-node-resolver/0.log" Jul 09 21:55:23.432229 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:23.432201 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dp58g_95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b/node-ca/0.log" Jul 09 21:55:23.595522 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:23.595480 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:23.595522 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:23.595522 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:23.595907 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:23.595614 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 21:55:23.595907 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:23.595630 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 21:55:23.595907 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:23.595678 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:27.595661715 +0000 UTC m=+246.906279216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : configmap "kube-rbac-proxy" not found Jul 09 21:55:23.595907 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:23.595692 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:27.595686405 +0000 UTC m=+246.906303905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : secret "insights-runtime-extractor-tls" not found Jul 09 21:55:24.231176 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:24.231147 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8n2cw_115dcab9-3d51-43fa-ab57-c00a61331fc3/serve-healthcheck-canary/0.log" Jul 09 21:55:25.076487 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:25.076445 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:25.076982 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:25.076507 2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:25.076982 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:25.076914 2556 scope.go:117] "RemoveContainer" containerID="18ad990c4e3facc040f9679dfa390cf156268bf16c1cac3bfa5ee54335f1cc8d" Jul 09 21:55:25.077118 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:25.077095 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-xfqh4_openshift-console-operator(4f715282-1ab7-427e-973a-550762cade1e)\"" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" podUID="4f715282-1ab7-427e-973a-550762cade1e" Jul 09 21:55:25.233367 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:25.233334 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-6kzqk_520a37a8-e07d-41c7-99c5-51bcbdd6ef23/kube-storage-version-migrator-operator/0.log" Jul 09 21:55:27.627795 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:27.627751 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:27.627795 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:27.627796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:27.628207 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:27.627875 2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 21:55:27.628207 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:27.627907 2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 21:55:27.628207 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:27.627945 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:35.627925294 +0000 UTC m=+254.938542809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : configmap "kube-rbac-proxy" not found Jul 09 21:55:27.628207 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:27.627960 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls podName:d5894dba-53ef-4271-838c-3420e112755b nodeName:}" failed. No retries permitted until 2026-07-09 21:55:35.62795356 +0000 UTC m=+254.938571063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2677l" (UID: "d5894dba-53ef-4271-838c-3420e112755b") : secret "insights-runtime-extractor-tls" not found Jul 09 21:55:30.553165 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.553133 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:30.555642 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.555619 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3563e31a-0ecb-4d93-8cc0-c393645a63e2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b2m26\" (UID: \"3563e31a-0ecb-4d93-8cc0-c393645a63e2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:30.591102 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.591073 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-9hrd7\"" Jul 09 21:55:30.599073 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.599054 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" Jul 09 21:55:30.654156 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.654122 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:30.654327 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.654209 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:30.654866 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.654827 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-service-ca-bundle\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:30.657077 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.657051 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139e71b1-5134-4bc2-8a29-96f5dc16c5b6-metrics-certs\") pod \"router-default-6b5486fbd7-n8ft6\" (UID: \"139e71b1-5134-4bc2-8a29-96f5dc16c5b6\") " pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:30.683657 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.683634 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k7jf4\"" Jul 09 21:55:30.691900 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.691838 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:30.720023 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.719979 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26"] Jul 09 21:55:30.818020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:30.817946 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b5486fbd7-n8ft6"] Jul 09 21:55:30.820542 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:30.820516 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139e71b1_5134_4bc2_8a29_96f5dc16c5b6.slice/crio-3c3a672429bb05feb3b86497e94b9c838628bf0be062578a1c9259ce82d2ea15 WatchSource:0}: Error finding container 3c3a672429bb05feb3b86497e94b9c838628bf0be062578a1c9259ce82d2ea15: Status 404 returned error can't find the container with id 3c3a672429bb05feb3b86497e94b9c838628bf0be062578a1c9259ce82d2ea15 Jul 09 21:55:31.139373 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:31.139345 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" event={"ID":"3563e31a-0ecb-4d93-8cc0-c393645a63e2","Type":"ContainerStarted","Data":"bee17bc4ff17c303d759a0d4b1e70868f76fee3e1df697c039dc061b06f014c5"} Jul 09 21:55:31.140630 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:31.140608 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" event={"ID":"139e71b1-5134-4bc2-8a29-96f5dc16c5b6","Type":"ContainerStarted","Data":"bd0ae6db48cf91be13a34245e0d34fc489158e34d9647d5b3a8938054bdd9493"} Jul 09 21:55:31.140719 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:31.140652 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" event={"ID":"139e71b1-5134-4bc2-8a29-96f5dc16c5b6","Type":"ContainerStarted","Data":"3c3a672429bb05feb3b86497e94b9c838628bf0be062578a1c9259ce82d2ea15"} Jul 09 21:55:31.158723 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:31.158679 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" podStartSLOduration=17.1586658 podStartE2EDuration="17.1586658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:31.157953817 +0000 UTC m=+250.468571351" watchObservedRunningTime="2026-07-09 21:55:31.1586658 +0000 UTC m=+250.469283322" Jul 09 21:55:31.692196 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:31.692170 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:31.695134 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:31.695108 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:32.145290 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:32.145256 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:32.146737 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:32.146710 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b5486fbd7-n8ft6" Jul 09 21:55:32.975121 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:32.975098 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:55:32.977177 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:32.977158 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceeea23b-5ed9-45c1-97c0-c223a39f03b9-metrics-certs\") pod \"network-metrics-daemon-5jcvb\" (UID: \"ceeea23b-5ed9-45c1-97c0-c223a39f03b9\") " pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:55:33.149710 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:33.149673 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" event={"ID":"3563e31a-0ecb-4d93-8cc0-c393645a63e2","Type":"ContainerStarted","Data":"b1e2f635b4afd1c8d18a9f52556575f7fe8b57fa84680300158a4e19d0b29a31"} Jul 09 21:55:33.150035 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:33.149732 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" event={"ID":"3563e31a-0ecb-4d93-8cc0-c393645a63e2","Type":"ContainerStarted","Data":"48a64e841441c9170f25763ba8c4f23cc16348b1ff191e693a78a4ed51b015e2"} Jul 09 21:55:33.167158 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:33.167102 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b2m26" podStartSLOduration=16.975856653 podStartE2EDuration="19.167084054s" totalImagesPullingTime="2.191227401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:33.166499828 +0000 UTC m=+252.477117352" watchObservedRunningTime="2026-07-09 21:55:33.167084054 +0000 UTC m=+252.477701576" Jul 09 21:55:33.173694 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:33.173672 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdkql\"" Jul 09 21:55:33.181776 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:33.181757 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5jcvb" Jul 09 21:55:33.307248 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:33.307219 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5jcvb"] Jul 09 21:55:33.310277 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:33.310246 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceeea23b_5ed9_45c1_97c0_c223a39f03b9.slice/crio-6673ae5c33a7318723016f95b1c08e314e3bad1f87be267c80248e80af61b208 WatchSource:0}: Error finding container 6673ae5c33a7318723016f95b1c08e314e3bad1f87be267c80248e80af61b208: Status 404 returned error can't find the container with id 6673ae5c33a7318723016f95b1c08e314e3bad1f87be267c80248e80af61b208 Jul 09 21:55:34.153664 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:34.153627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5jcvb" event={"ID":"ceeea23b-5ed9-45c1-97c0-c223a39f03b9","Type":"ContainerStarted","Data":"6673ae5c33a7318723016f95b1c08e314e3bad1f87be267c80248e80af61b208"} Jul 09 21:55:35.158343 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.158310 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5jcvb" event={"ID":"ceeea23b-5ed9-45c1-97c0-c223a39f03b9","Type":"ContainerStarted","Data":"8f6585ddebd45e2c1ba4cfa1a7e09453725b6754ac7b964f333defc7d64a6f04"} Jul 09 21:55:35.158725 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.158371 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5jcvb" event={"ID":"ceeea23b-5ed9-45c1-97c0-c223a39f03b9","Type":"ContainerStarted","Data":"ec2a49c911100cda513ce56b68517688bf276d459551c3f58223eb82f67e6215"} Jul 09 21:55:35.176090 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.176051 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5jcvb" podStartSLOduration=252.928355184 podStartE2EDuration="4m14.176037772s" totalImagesPullingTime="1.247682588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:35.175058533 +0000 UTC m=+254.485676056" watchObservedRunningTime="2026-07-09 21:55:35.176037772 +0000 UTC m=+254.486655294" Jul 09 21:55:35.696518 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.696486 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:35.696695 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.696525 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:35.697071 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.697034 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5894dba-53ef-4271-838c-3420e112755b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:35.698859 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.698840 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5894dba-53ef-4271-838c-3420e112755b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2677l\" (UID: \"d5894dba-53ef-4271-838c-3420e112755b\") " pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:35.760513 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.760481 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-78g6p\"" Jul 09 21:55:35.768517 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.768496 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2677l" Jul 09 21:55:35.892064 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:35.892029 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2677l"] Jul 09 21:55:35.894829 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:35.894792 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5894dba_53ef_4271_838c_3420e112755b.slice/crio-4c5aca27e6e8ed17bd494e407d57659b85026bdca9b9fc39b40d3d9e5aa1f4c6 WatchSource:0}: Error finding container 4c5aca27e6e8ed17bd494e407d57659b85026bdca9b9fc39b40d3d9e5aa1f4c6: Status 404 returned error can't find the container with id 4c5aca27e6e8ed17bd494e407d57659b85026bdca9b9fc39b40d3d9e5aa1f4c6 Jul 09 21:55:36.162938 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:36.162900 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2677l" event={"ID":"d5894dba-53ef-4271-838c-3420e112755b","Type":"ContainerStarted","Data":"4c5aca27e6e8ed17bd494e407d57659b85026bdca9b9fc39b40d3d9e5aa1f4c6"} Jul 09 21:55:38.270609 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:38.270560 2556 scope.go:117] "RemoveContainer" containerID="18ad990c4e3facc040f9679dfa390cf156268bf16c1cac3bfa5ee54335f1cc8d" Jul 09 21:55:40.179728 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:40.179686 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 21:55:40.180136 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:40.179823 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" event={"ID":"4f715282-1ab7-427e-973a-550762cade1e","Type":"ContainerStarted","Data":"46e7e03e384af3ebb8b1881117bda982df459c9de74beb869fbf0b24920ca031"} Jul 09 21:55:40.180268 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:40.180251 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:40.182310 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:40.182285 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2677l" event={"ID":"d5894dba-53ef-4271-838c-3420e112755b","Type":"ContainerStarted","Data":"afeb8db7822c49fa0ba76bfbb4caf48b968767e61d68c4535507e4b80fbd63d4"} Jul 09 21:55:40.197351 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:40.197301 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" podStartSLOduration=23.123764191 podStartE2EDuration="26.197280963s" totalImagesPullingTime="3.073516772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:40.195714471 +0000 UTC m=+259.506331992" watchObservedRunningTime="2026-07-09 21:55:40.197280963 +0000 UTC m=+259.507898489" Jul 09 21:55:40.363554 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:40.363518 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-xfqh4" Jul 09 21:55:41.189217 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.189185 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2677l" event={"ID":"d5894dba-53ef-4271-838c-3420e112755b","Type":"ContainerStarted","Data":"c09a2deafb19886a7ce7596019c338461570399334e1758c9bd9feca0ebbff3d"} Jul 09 21:55:41.706671 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.706456 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-86lx5"] Jul 09 21:55:41.711031 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.711008 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-86lx5" Jul 09 21:55:41.713998 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.713973 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 21:55:41.714140 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.714062 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 21:55:41.714313 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.714293 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tm7f9\"" Jul 09 21:55:41.719680 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.719658 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-86lx5"] Jul 09 21:55:41.744651 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.744442 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxjcg\" (UniqueName: \"kubernetes.io/projected/25c8a8d3-26e2-484c-ba48-0b79b989ee38-kube-api-access-cxjcg\") pod \"downloads-685bfb5f64-86lx5\" (UID: \"25c8a8d3-26e2-484c-ba48-0b79b989ee38\") " pod="openshift-console/downloads-685bfb5f64-86lx5" Jul 09 21:55:41.746169 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.746145 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-678cb6686-wvkh5"] Jul 09 21:55:41.748470 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.748447 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.751807 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.751786 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 21:55:41.752175 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.752054 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 21:55:41.752175 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.752126 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hx9tw\"" Jul 09 21:55:41.752589 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.752557 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 21:55:41.760570 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.760552 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 21:55:41.772504 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.772477 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678cb6686-wvkh5"] Jul 09 21:55:41.846420 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846361 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-registry-tls\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.846420 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846411 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30f64e04-7199-470a-86ca-cd96a6fb3021-image-registry-private-configuration\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.846702 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846443 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30f64e04-7199-470a-86ca-cd96a6fb3021-ca-trust-extracted\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.846702 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846487 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-bound-sa-token\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.846702 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846523 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjcg\" (UniqueName: \"kubernetes.io/projected/25c8a8d3-26e2-484c-ba48-0b79b989ee38-kube-api-access-cxjcg\") pod \"downloads-685bfb5f64-86lx5\" (UID: \"25c8a8d3-26e2-484c-ba48-0b79b989ee38\") " pod="openshift-console/downloads-685bfb5f64-86lx5" Jul 09 21:55:41.846702 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846595 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnqb\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-kube-api-access-zlnqb\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.846702 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846641 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30f64e04-7199-470a-86ca-cd96a6fb3021-registry-certificates\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.846702 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846697 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30f64e04-7199-470a-86ca-cd96a6fb3021-installation-pull-secrets\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.846958 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.846726 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30f64e04-7199-470a-86ca-cd96a6fb3021-trusted-ca\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.857509 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.857481 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjcg\" (UniqueName: \"kubernetes.io/projected/25c8a8d3-26e2-484c-ba48-0b79b989ee38-kube-api-access-cxjcg\") pod \"downloads-685bfb5f64-86lx5\" (UID: \"25c8a8d3-26e2-484c-ba48-0b79b989ee38\") " pod="openshift-console/downloads-685bfb5f64-86lx5" Jul 09 21:55:41.948086 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.947997 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30f64e04-7199-470a-86ca-cd96a6fb3021-registry-certificates\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.948086 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948066 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30f64e04-7199-470a-86ca-cd96a6fb3021-installation-pull-secrets\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.948305 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948098 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30f64e04-7199-470a-86ca-cd96a6fb3021-trusted-ca\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.948305 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948151 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-registry-tls\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.948305 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948183 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30f64e04-7199-470a-86ca-cd96a6fb3021-image-registry-private-configuration\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.948305 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948212 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30f64e04-7199-470a-86ca-cd96a6fb3021-ca-trust-extracted\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.948305 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948253 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-bound-sa-token\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.948305 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948297 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnqb\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-kube-api-access-zlnqb\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.949021 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948732 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/30f64e04-7199-470a-86ca-cd96a6fb3021-ca-trust-extracted\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.949021 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.948943 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/30f64e04-7199-470a-86ca-cd96a6fb3021-registry-certificates\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.949489 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.949466 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30f64e04-7199-470a-86ca-cd96a6fb3021-trusted-ca\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.951124 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.951079 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/30f64e04-7199-470a-86ca-cd96a6fb3021-installation-pull-secrets\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.951236 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.951145 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/30f64e04-7199-470a-86ca-cd96a6fb3021-image-registry-private-configuration\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.951307 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.951290 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-registry-tls\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.956794 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.956762 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-bound-sa-token\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:41.956794 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:41.956778 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnqb\" (UniqueName: \"kubernetes.io/projected/30f64e04-7199-470a-86ca-cd96a6fb3021-kube-api-access-zlnqb\") pod \"image-registry-678cb6686-wvkh5\" (UID: \"30f64e04-7199-470a-86ca-cd96a6fb3021\") " pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:42.023650 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:42.023603 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-86lx5" Jul 09 21:55:42.060283 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:42.060248 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:42.795986 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:42.795965 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-86lx5"] Jul 09 21:55:42.797665 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:42.797641 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c8a8d3_26e2_484c_ba48_0b79b989ee38.slice/crio-683e9f7558fb20bfd6949649bfd91e23666a492ea548b7560df647e0edc8bd9d WatchSource:0}: Error finding container 683e9f7558fb20bfd6949649bfd91e23666a492ea548b7560df647e0edc8bd9d: Status 404 returned error can't find the container with id 683e9f7558fb20bfd6949649bfd91e23666a492ea548b7560df647e0edc8bd9d Jul 09 21:55:42.809214 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:42.809187 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-678cb6686-wvkh5"] Jul 09 21:55:42.811734 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:42.811709 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f64e04_7199_470a_86ca_cd96a6fb3021.slice/crio-1606167d58c4c0056c6ddb146ac45f07c3c43a4cc280065170981cb01152efdd WatchSource:0}: Error finding container 1606167d58c4c0056c6ddb146ac45f07c3c43a4cc280065170981cb01152efdd: Status 404 returned error can't find the container with id 1606167d58c4c0056c6ddb146ac45f07c3c43a4cc280065170981cb01152efdd Jul 09 21:55:43.198227 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:43.198193 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2677l" event={"ID":"d5894dba-53ef-4271-838c-3420e112755b","Type":"ContainerStarted","Data":"a67dfaff61a44494f4fbf5e1d76bd2cc5d618f66f10af21ae1bf15cd164a1d12"} Jul 09 21:55:43.199407 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:43.199381 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-86lx5" event={"ID":"25c8a8d3-26e2-484c-ba48-0b79b989ee38","Type":"ContainerStarted","Data":"683e9f7558fb20bfd6949649bfd91e23666a492ea548b7560df647e0edc8bd9d"} Jul 09 21:55:43.200863 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:43.200829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678cb6686-wvkh5" event={"ID":"30f64e04-7199-470a-86ca-cd96a6fb3021","Type":"ContainerStarted","Data":"a35c51bd26f012470991b604004937b3f94d50730e44b3fd48117200b809ee0d"} Jul 09 21:55:43.200969 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:43.200873 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-678cb6686-wvkh5" event={"ID":"30f64e04-7199-470a-86ca-cd96a6fb3021","Type":"ContainerStarted","Data":"1606167d58c4c0056c6ddb146ac45f07c3c43a4cc280065170981cb01152efdd"} Jul 09 21:55:43.200969 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:43.200949 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:55:43.215321 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:43.215054 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2677l" podStartSLOduration=17.565723248 podStartE2EDuration="24.215033692s" totalImagesPullingTime="6.649310444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 21:55:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:43.213893184 +0000 UTC m=+262.524510718" watchObservedRunningTime="2026-07-09 21:55:43.215033692 +0000 UTC m=+262.525651215" Jul 09 21:55:43.231153 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:43.231088 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-678cb6686-wvkh5" podStartSLOduration=2.231067267 podStartE2EDuration="2.231067267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:43.230455928 +0000 UTC m=+262.541073453" watchObservedRunningTime="2026-07-09 21:55:43.231067267 +0000 UTC m=+262.541684791" Jul 09 21:55:46.277071 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.277039 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9"] Jul 09 21:55:46.282521 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.282498 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.286304 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.286276 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 21:55:46.286427 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.286328 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 09 21:55:46.286752 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.286611 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 21:55:46.286752 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.286650 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 09 21:55:46.286752 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.286698 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x7pf2\"" Jul 09 21:55:46.286982 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.286888 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 21:55:46.290385 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.290363 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9"] Jul 09 21:55:46.389185 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.389155 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117fe50e-09df-4baa-99f5-cbb6f561d195-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.389365 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.389227 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.389365 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.389290 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.389365 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.389348 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvfcg\" (UniqueName: \"kubernetes.io/projected/117fe50e-09df-4baa-99f5-cbb6f561d195-kube-api-access-gvfcg\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.490649 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.490614 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.490828 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.490659 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.490828 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.490680 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvfcg\" (UniqueName: \"kubernetes.io/projected/117fe50e-09df-4baa-99f5-cbb6f561d195-kube-api-access-gvfcg\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.490828 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.490726 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117fe50e-09df-4baa-99f5-cbb6f561d195-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.490828 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:46.490791 2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 09 21:55:46.491038 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:46.490877 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-tls podName:117fe50e-09df-4baa-99f5-cbb6f561d195 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:46.990856755 +0000 UTC m=+266.301474261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-jp4f9" (UID: "117fe50e-09df-4baa-99f5-cbb6f561d195") : secret "prometheus-operator-tls" not found Jul 09 21:55:46.491461 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.491440 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/117fe50e-09df-4baa-99f5-cbb6f561d195-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.493681 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.493658 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.501844 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.501825 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvfcg\" (UniqueName: \"kubernetes.io/projected/117fe50e-09df-4baa-99f5-cbb6f561d195-kube-api-access-gvfcg\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.994940 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.994905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:46.997744 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:46.997718 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/117fe50e-09df-4baa-99f5-cbb6f561d195-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-jp4f9\" (UID: \"117fe50e-09df-4baa-99f5-cbb6f561d195\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:47.194590 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:47.194535 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" Jul 09 21:55:47.337671 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:47.337616 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9"] Jul 09 21:55:47.339864 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:47.339822 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117fe50e_09df_4baa_99f5_cbb6f561d195.slice/crio-0b1529e60635a864bb7ca1a6f346c7c3242915fe6a8aea278c78e07c55f9dc6b WatchSource:0}: Error finding container 0b1529e60635a864bb7ca1a6f346c7c3242915fe6a8aea278c78e07c55f9dc6b: Status 404 returned error can't find the container with id 0b1529e60635a864bb7ca1a6f346c7c3242915fe6a8aea278c78e07c55f9dc6b Jul 09 21:55:48.220509 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:48.220429 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" event={"ID":"117fe50e-09df-4baa-99f5-cbb6f561d195","Type":"ContainerStarted","Data":"0b1529e60635a864bb7ca1a6f346c7c3242915fe6a8aea278c78e07c55f9dc6b"} Jul 09 21:55:49.228137 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:49.228094 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" event={"ID":"117fe50e-09df-4baa-99f5-cbb6f561d195","Type":"ContainerStarted","Data":"face18750120e21a022ed7d1a6320d14276fa46df53f7a91625fe3bd7c411de2"} Jul 09 21:55:50.233993 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:50.233950 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" event={"ID":"117fe50e-09df-4baa-99f5-cbb6f561d195","Type":"ContainerStarted","Data":"fe4aa8c0bc7aff2b0e7042baa032fd97467a8c44bf295479947c133d39628aef"} Jul 09 21:55:50.251503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:50.251449 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-jp4f9" podStartSLOduration=2.487880324 podStartE2EDuration="4.251429088s" totalImagesPullingTime="1.763548764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:55:50.250503737 +0000 UTC m=+269.561121262" watchObservedRunningTime="2026-07-09 21:55:50.251429088 +0000 UTC m=+269.562046593" Jul 09 21:55:52.626781 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.626747 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6"] Jul 09 21:55:52.644717 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.644682 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6"] Jul 09 21:55:52.644889 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.644802 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.645895 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.645871 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7f248"] Jul 09 21:55:52.647617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.647590 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 09 21:55:52.647736 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.647620 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 09 21:55:52.647736 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.647566 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-78xg5\"" Jul 09 21:55:52.664992 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.664973 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.667548 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.667511 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 21:55:52.667678 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.667557 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 21:55:52.667678 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.667565 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7bmbr\"" Jul 09 21:55:52.667678 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.667671 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 21:55:52.739070 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739034 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c48e0e65-afa9-4b13-9992-c575253f6260-metrics-client-ca\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.739258 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739077 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-root\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.739258 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739107 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.739258 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739132 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxxbp\" (UniqueName: \"kubernetes.io/projected/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-kube-api-access-jxxbp\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.739670 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739645 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-wtmp\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.739772 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739677 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7jz\" (UniqueName: \"kubernetes.io/projected/c48e0e65-afa9-4b13-9992-c575253f6260-kube-api-access-zm7jz\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.739772 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739695 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.739772 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739722 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-sys\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.739932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739768 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-accelerators-collector-config\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.739932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739813 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.739932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739859 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.739932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739885 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-tls\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.739932 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.739928 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-textfile\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841212 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841166 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.841212 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841205 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-tls\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841459 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841227 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-textfile\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841459 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841251 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c48e0e65-afa9-4b13-9992-c575253f6260-metrics-client-ca\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841459 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-root\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841459 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:52.841365 2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 21:55:52.841459 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841398 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.841747 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:52.841627 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-tls podName:c48e0e65-afa9-4b13-9992-c575253f6260 nodeName:}" failed. No retries permitted until 2026-07-09 21:55:53.341415513 +0000 UTC m=+272.652033019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-tls") pod "node-exporter-7f248" (UID: "c48e0e65-afa9-4b13-9992-c575253f6260") : secret "node-exporter-tls" not found Jul 09 21:55:52.841747 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841692 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-root\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841747 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841632 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-textfile\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841987 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841674 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxxbp\" (UniqueName: \"kubernetes.io/projected/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-kube-api-access-jxxbp\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.841987 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841842 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-wtmp\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841987 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841884 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7jz\" (UniqueName: \"kubernetes.io/projected/c48e0e65-afa9-4b13-9992-c575253f6260-kube-api-access-zm7jz\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841987 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841915 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.841987 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841954 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-sys\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.841987 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841978 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-accelerators-collector-config\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.842356 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.841999 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c48e0e65-afa9-4b13-9992-c575253f6260-metrics-client-ca\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.842356 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.842035 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-wtmp\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.842356 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.842004 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.842356 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:52.842081 2556 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 09 21:55:52.842356 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.842055 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.842356 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:52.842134 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-tls podName:f51bbc95-3cb3-4f86-b1b2-de3022e7126d nodeName:}" failed. No retries permitted until 2026-07-09 21:55:53.34211833 +0000 UTC m=+272.652735833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-98pn6" (UID: "f51bbc95-3cb3-4f86-b1b2-de3022e7126d") : secret "openshift-state-metrics-tls" not found Jul 09 21:55:52.842356 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.842138 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c48e0e65-afa9-4b13-9992-c575253f6260-sys\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.842774 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.842512 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-accelerators-collector-config\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.844533 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.844509 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.844669 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.844551 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:52.853842 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.853817 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7jz\" (UniqueName: \"kubernetes.io/projected/c48e0e65-afa9-4b13-9992-c575253f6260-kube-api-access-zm7jz\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:52.854213 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:52.854184 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxxbp\" (UniqueName: \"kubernetes.io/projected/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-kube-api-access-jxxbp\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:53.348541 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.348490 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:53.348734 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.348606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-tls\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:53.351200 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.351175 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c48e0e65-afa9-4b13-9992-c575253f6260-node-exporter-tls\") pod \"node-exporter-7f248\" (UID: \"c48e0e65-afa9-4b13-9992-c575253f6260\") " pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:53.351325 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.351198 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f51bbc95-3cb3-4f86-b1b2-de3022e7126d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-98pn6\" (UID: \"f51bbc95-3cb3-4f86-b1b2-de3022e7126d\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:53.555114 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.555077 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" Jul 09 21:55:53.576464 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.576430 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7f248" Jul 09 21:55:53.742355 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.742318 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:55:53.750049 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.750023 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.752687 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.752499 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.753569 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.753874 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.754058 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.754116 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.754240 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kssn8\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.754286 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.754459 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.754558 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 21:55:53.755046 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.754060 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 21:55:53.760989 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.760949 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:55:53.854173 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854136 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-web-config\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854346 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854192 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854346 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854258 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854346 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854290 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854346 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854330 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854535 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854416 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854535 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854433 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854535 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854473 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854535 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854493 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-kube-api-access-4sh74\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854535 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854521 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854767 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854567 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-volume\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854767 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854609 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-out\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.854767 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.854638 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.955082 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955002 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-web-config\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.955082 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955039 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.955082 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955059 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.955082 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955077 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.955377 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:53.955207 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle podName:58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc nodeName:}" failed. No retries permitted until 2026-07-09 21:55:54.455185856 +0000 UTC m=+273.765803371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc") : configmap references non-existent config key: ca-bundle.crt Jul 09 21:55:53.955469 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955548 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955606 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955640 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955652 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955701 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-kube-api-access-4sh74\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:53.955706 2556 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:55:53.955758 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls podName:58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc nodeName:}" failed. No retries permitted until 2026-07-09 21:55:54.455740856 +0000 UTC m=+273.766358358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc") : secret "alertmanager-main-tls" not found Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955787 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955822 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-volume\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955847 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-out\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.956412 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.955874 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.957047 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.956967 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.958948 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.958921 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.959047 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.958931 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-web-config\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.959047 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.958960 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-out\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.959911 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.959868 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-volume\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.960330 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.960286 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.960465 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.960447 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.960664 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.960645 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.960948 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.960923 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:53.964346 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:53.964326 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-kube-api-access-4sh74\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:54.460485 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:54.460445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:54.460691 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:54.460556 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:54.463727 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:54.463677 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:54.463931 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:54.463902 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:54.663509 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:54.663473 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:55:55.626947 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.626916 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5457574644-9b7q9"] Jul 09 21:55:55.645168 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.645135 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5457574644-9b7q9"] Jul 09 21:55:55.645314 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.645261 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.648096 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.648065 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 09 21:55:55.648323 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.648303 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6hzsr\"" Jul 09 21:55:55.648428 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.648322 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 09 21:55:55.648552 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.648536 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 09 21:55:55.648769 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.648750 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ckqmga62ttt6m\"" Jul 09 21:55:55.648887 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.648811 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 09 21:55:55.648951 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.648937 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 09 21:55:55.771810 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.771781 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.772011 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.771832 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-grpc-tls\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.772011 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.771894 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-tls\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.772011 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.771929 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.772154 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.772027 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.772154 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.772064 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.772154 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.772089 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b054f-21a3-404c-967a-1fd09a092cda-metrics-client-ca\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.772349 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.772159 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7p9g\" (UniqueName: \"kubernetes.io/projected/5f9b054f-21a3-404c-967a-1fd09a092cda-kube-api-access-h7p9g\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.872664 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.872629 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.872664 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.872676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-grpc-tls\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.872918 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.872709 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-tls\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.872918 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.872734 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.872918 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.872775 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.872918 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.872799 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.872918 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.872822 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b054f-21a3-404c-967a-1fd09a092cda-metrics-client-ca\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.872918 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.872897 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7p9g\" (UniqueName: \"kubernetes.io/projected/5f9b054f-21a3-404c-967a-1fd09a092cda-kube-api-access-h7p9g\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.873958 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.873915 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b054f-21a3-404c-967a-1fd09a092cda-metrics-client-ca\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.875888 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.875843 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-grpc-tls\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.875888 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.875861 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.876071 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.876026 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.876169 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.876151 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.876651 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.876632 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-tls\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.876759 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.876717 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f9b054f-21a3-404c-967a-1fd09a092cda-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.884713 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.884661 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7p9g\" (UniqueName: \"kubernetes.io/projected/5f9b054f-21a3-404c-967a-1fd09a092cda-kube-api-access-h7p9g\") pod \"thanos-querier-5457574644-9b7q9\" (UID: \"5f9b054f-21a3-404c-967a-1fd09a092cda\") " pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:55.957764 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:55:55.957722 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:55:59.885286 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:55:59.885220 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48e0e65_afa9_4b13_9992_c575253f6260.slice/crio-6b23808cd9f4a8b393731d0a52b094cd39e5a2a53b97bd8565cb1e37f2de28fa WatchSource:0}: Error finding container 6b23808cd9f4a8b393731d0a52b094cd39e5a2a53b97bd8565cb1e37f2de28fa: Status 404 returned error can't find the container with id 6b23808cd9f4a8b393731d0a52b094cd39e5a2a53b97bd8565cb1e37f2de28fa Jul 09 21:56:00.027523 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:00.027496 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6"] Jul 09 21:56:00.176695 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:56:00.176618 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51bbc95_3cb3_4f86_b1b2_de3022e7126d.slice/crio-c8390de730d042a301ada02bfa2d76e075684c9c7ff275e4c8938b77692e57c1 WatchSource:0}: Error finding container c8390de730d042a301ada02bfa2d76e075684c9c7ff275e4c8938b77692e57c1: Status 404 returned error can't find the container with id c8390de730d042a301ada02bfa2d76e075684c9c7ff275e4c8938b77692e57c1 Jul 09 21:56:00.254680 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:00.254647 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5457574644-9b7q9"] Jul 09 21:56:00.258488 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:56:00.258460 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9b054f_21a3_404c_967a_1fd09a092cda.slice/crio-f51ce748c6b3c766e2e61be938802ccdf8e8439af855655d90209e824c9602cf WatchSource:0}: Error finding container f51ce748c6b3c766e2e61be938802ccdf8e8439af855655d90209e824c9602cf: Status 404 returned error can't find the container with id f51ce748c6b3c766e2e61be938802ccdf8e8439af855655d90209e824c9602cf Jul 09 21:56:00.258862 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:00.258838 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:56:00.262405 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:56:00.262379 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ab1fda_f579_4078_b8fe_c9f5b5eeb4fc.slice/crio-d32b335c5cfb675b11eb9d699142a6535de60be4c260dd7eb80bec2b35647bfc WatchSource:0}: Error finding container d32b335c5cfb675b11eb9d699142a6535de60be4c260dd7eb80bec2b35647bfc: Status 404 returned error can't find the container with id d32b335c5cfb675b11eb9d699142a6535de60be4c260dd7eb80bec2b35647bfc Jul 09 21:56:00.273923 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:00.273896 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7f248" event={"ID":"c48e0e65-afa9-4b13-9992-c575253f6260","Type":"ContainerStarted","Data":"6b23808cd9f4a8b393731d0a52b094cd39e5a2a53b97bd8565cb1e37f2de28fa"} Jul 09 21:56:00.274990 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:00.274968 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" event={"ID":"5f9b054f-21a3-404c-967a-1fd09a092cda","Type":"ContainerStarted","Data":"f51ce748c6b3c766e2e61be938802ccdf8e8439af855655d90209e824c9602cf"} Jul 09 21:56:00.276171 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:00.276147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerStarted","Data":"d32b335c5cfb675b11eb9d699142a6535de60be4c260dd7eb80bec2b35647bfc"} Jul 09 21:56:00.277447 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:00.277423 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" event={"ID":"f51bbc95-3cb3-4f86-b1b2-de3022e7126d","Type":"ContainerStarted","Data":"c8390de730d042a301ada02bfa2d76e075684c9c7ff275e4c8938b77692e57c1"} Jul 09 21:56:01.286863 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:01.286053 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-86lx5" event={"ID":"25c8a8d3-26e2-484c-ba48-0b79b989ee38","Type":"ContainerStarted","Data":"1263a5e4d66a905649181ad2cfa5873fba80ea23910723daf9e2bff3d54d89f1"} Jul 09 21:56:01.286863 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:01.286620 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-86lx5" Jul 09 21:56:01.290132 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:01.290043 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" event={"ID":"f51bbc95-3cb3-4f86-b1b2-de3022e7126d","Type":"ContainerStarted","Data":"4372173f06de6cc79f6b4fe79a3f0ce57c113923906fcb4b3ba814ffce09ab64"} Jul 09 21:56:01.290132 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:01.290094 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" event={"ID":"f51bbc95-3cb3-4f86-b1b2-de3022e7126d","Type":"ContainerStarted","Data":"56d968a47bbc3b9cc917e1a385c4c17fe9c21aeb9b5f9946edb3ab86bfb6b6f8"} Jul 09 21:56:01.293428 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:01.293236 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-86lx5" Jul 09 21:56:01.308980 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:01.307871 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-86lx5" podStartSLOduration=2.43458623 podStartE2EDuration="20.307851919s" totalImagesPullingTime="17.873265689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:56:01.30586389 +0000 UTC m=+280.616481447" watchObservedRunningTime="2026-07-09 21:56:01.307851919 +0000 UTC m=+280.618469447" Jul 09 21:56:02.297754 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:02.297556 2556 generic.go:356] "Generic (PLEG): container finished" podID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerID="c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c" exitCode=0 Jul 09 21:56:02.297754 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:02.297627 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerDied","Data":"c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c"} Jul 09 21:56:02.302921 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:02.302888 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" event={"ID":"f51bbc95-3cb3-4f86-b1b2-de3022e7126d","Type":"ContainerStarted","Data":"e127e73eb01a4bdedc428e076f4133bf40b0073894c62b1ba4b8bac5d52ebc91"} Jul 09 21:56:02.305964 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:02.305845 2556 generic.go:356] "Generic (PLEG): container finished" podID="c48e0e65-afa9-4b13-9992-c575253f6260" containerID="126835c0342873d3f020654b90ee4f1b62b2b2686c6d321e0a202c188c1197a5" exitCode=0 Jul 09 21:56:02.306794 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:02.306616 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7f248" event={"ID":"c48e0e65-afa9-4b13-9992-c575253f6260","Type":"ContainerDied","Data":"126835c0342873d3f020654b90ee4f1b62b2b2686c6d321e0a202c188c1197a5"} Jul 09 21:56:02.368133 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:02.367855 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-98pn6" podStartSLOduration=8.943850572 podStartE2EDuration="10.367832916s" totalImagesPullingTime="1.423982344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:56:02.364714392 +0000 UTC m=+281.675331916" watchObservedRunningTime="2026-07-09 21:56:02.367832916 +0000 UTC m=+281.678450442" Jul 09 21:56:02.895835 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:56:02.895764 2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-b4sg8" podUID="4488e1dc-e74a-48a7-a208-e6371f9d4375" Jul 09 21:56:03.314482 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:03.314458 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4sg8" Jul 09 21:56:03.314921 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:03.314501 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7f248" event={"ID":"c48e0e65-afa9-4b13-9992-c575253f6260","Type":"ContainerStarted","Data":"848f491e2f408eb3472db3634b51e38277335377a3edbc278cedb5227493606f"} Jul 09 21:56:03.314921 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:03.314548 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7f248" event={"ID":"c48e0e65-afa9-4b13-9992-c575253f6260","Type":"ContainerStarted","Data":"07dcffd69babf5a54bccc6c92d442a7105c966c0dbdf5bc1135528f8b7949250"} Jul 09 21:56:03.335999 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:03.335859 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7f248" podStartSLOduration=9.716510423999999 podStartE2EDuration="11.335835959s" totalImagesPullingTime="1.517040271s" totalInitContainerRuntime="102.285264ms" isStatefulPod=true podCreationTimestamp="2026-07-09 21:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:56:03.335056224 +0000 UTC m=+282.645673748" watchObservedRunningTime="2026-07-09 21:56:03.335835959 +0000 UTC m=+282.646453486" Jul 09 21:56:04.210256 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:04.210225 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-678cb6686-wvkh5" Jul 09 21:56:04.325186 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:04.324509 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" event={"ID":"5f9b054f-21a3-404c-967a-1fd09a092cda","Type":"ContainerStarted","Data":"2e5eda9cb0cecc7d08cae17217963e8b27b51276e84dbc6b528794795937db42"} Jul 09 21:56:04.325186 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:04.324623 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" event={"ID":"5f9b054f-21a3-404c-967a-1fd09a092cda","Type":"ContainerStarted","Data":"10344a22126e52a8fb6da61752c094d1c7a87f5722b42fefed8f74a5353e66d9"} Jul 09 21:56:04.325186 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:04.324661 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" event={"ID":"5f9b054f-21a3-404c-967a-1fd09a092cda","Type":"ContainerStarted","Data":"a8ad12e2df55670b5d9ee47417b12d597280fb6198902628fe0ba78187b1f56f"} Jul 09 21:56:05.976749 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:05.976720 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:56:05.979183 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:05.979161 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4488e1dc-e74a-48a7-a208-e6371f9d4375-metrics-tls\") pod \"dns-default-b4sg8\" (UID: \"4488e1dc-e74a-48a7-a208-e6371f9d4375\") " pod="openshift-dns/dns-default-b4sg8" Jul 09 21:56:06.018403 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.018379 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4k4f4\"" Jul 09 21:56:06.025511 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.025429 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b4sg8" Jul 09 21:56:06.240833 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.240796 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b4sg8"] Jul 09 21:56:06.246856 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:56:06.246822 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4488e1dc_e74a_48a7_a208_e6371f9d4375.slice/crio-b492040ba48a8ff534b2fc118ad457ea6803cb8bdc8dd6f641bc95816d0f5058 WatchSource:0}: Error finding container b492040ba48a8ff534b2fc118ad457ea6803cb8bdc8dd6f641bc95816d0f5058: Status 404 returned error can't find the container with id b492040ba48a8ff534b2fc118ad457ea6803cb8bdc8dd6f641bc95816d0f5058 Jul 09 21:56:06.335929 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.335860 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4sg8" event={"ID":"4488e1dc-e74a-48a7-a208-e6371f9d4375","Type":"ContainerStarted","Data":"b492040ba48a8ff534b2fc118ad457ea6803cb8bdc8dd6f641bc95816d0f5058"} Jul 09 21:56:06.339750 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.339602 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" event={"ID":"5f9b054f-21a3-404c-967a-1fd09a092cda","Type":"ContainerStarted","Data":"0352a5c1a1771480bd429fdc2109e758eaa3a90e8c56f2acc5428fb4b84c4c96"} Jul 09 21:56:06.339750 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.339679 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" event={"ID":"5f9b054f-21a3-404c-967a-1fd09a092cda","Type":"ContainerStarted","Data":"52c356dcabaf2d621a9536da4430da7998ef82204050e386ce3939d297cbab45"} Jul 09 21:56:06.344354 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.344316 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerStarted","Data":"99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c"} Jul 09 21:56:06.344463 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.344394 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerStarted","Data":"27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec"} Jul 09 21:56:06.344463 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.344438 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerStarted","Data":"64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b"} Jul 09 21:56:06.344600 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.344482 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerStarted","Data":"0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560"} Jul 09 21:56:06.344600 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:06.344520 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerStarted","Data":"0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9"} Jul 09 21:56:07.355012 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:07.354972 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" event={"ID":"5f9b054f-21a3-404c-967a-1fd09a092cda","Type":"ContainerStarted","Data":"abb9e1c9e495c799b1f23a0ecd3ee9987e226b461af64cd0815b955511b70a17"} Jul 09 21:56:07.355557 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:07.355174 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:56:07.361313 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:07.361283 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerStarted","Data":"65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280"} Jul 09 21:56:07.382596 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:07.382535 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" podStartSLOduration=7.097909221 podStartE2EDuration="12.382519403s" totalImagesPullingTime="5.284610182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:56:07.380426789 +0000 UTC m=+286.691044353" watchObservedRunningTime="2026-07-09 21:56:07.382519403 +0000 UTC m=+286.693136995" Jul 09 21:56:07.412459 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:07.412351 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=10.13344893 podStartE2EDuration="14.412329909s" totalImagesPullingTime="4.253745176s" totalInitContainerRuntime="25.135803ms" isStatefulPod=false podCreationTimestamp="2026-07-09 21:55:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:56:07.409359072 +0000 UTC m=+286.719976595" watchObservedRunningTime="2026-07-09 21:56:07.412329909 +0000 UTC m=+286.722947434" Jul 09 21:56:08.378116 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:08.378089 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5457574644-9b7q9" Jul 09 21:56:09.375726 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:09.375685 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4sg8" event={"ID":"4488e1dc-e74a-48a7-a208-e6371f9d4375","Type":"ContainerStarted","Data":"83600be6a60903bdd93e83ea807de454956908a7b60d69fff95b899c292942b9"} Jul 09 21:56:09.375914 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:09.375750 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b4sg8" event={"ID":"4488e1dc-e74a-48a7-a208-e6371f9d4375","Type":"ContainerStarted","Data":"bb852892737407443b888077e5a660428001db0f05d0f6bbbd00ac9de1574526"} Jul 09 21:56:09.375914 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:09.375783 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b4sg8" Jul 09 21:56:09.394812 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:09.394763 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b4sg8" podStartSLOduration=251.927552481 podStartE2EDuration="4m14.394748181s" totalImagesPullingTime="2.4671957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:56:09.392800047 +0000 UTC m=+288.703417574" watchObservedRunningTime="2026-07-09 21:56:09.394748181 +0000 UTC m=+288.705365704" Jul 09 21:56:19.382500 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:19.382469 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b4sg8" Jul 09 21:56:21.098833 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:21.098802 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 21:56:21.100104 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:21.100058 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 21:56:21.110564 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:21.110547 2556 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 21:56:49.533837 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:49.533749 2556 generic.go:356] "Generic (PLEG): container finished" podID="520a37a8-e07d-41c7-99c5-51bcbdd6ef23" containerID="4903187f82747f3f110dd9fb84896848d25ee13612d70483146a96f8dcf10e41" exitCode=0 Jul 09 21:56:49.533837 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:49.533826 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" event={"ID":"520a37a8-e07d-41c7-99c5-51bcbdd6ef23","Type":"ContainerDied","Data":"4903187f82747f3f110dd9fb84896848d25ee13612d70483146a96f8dcf10e41"} Jul 09 21:56:49.534333 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:49.534175 2556 scope.go:117] "RemoveContainer" containerID="4903187f82747f3f110dd9fb84896848d25ee13612d70483146a96f8dcf10e41" Jul 09 21:56:49.535408 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:49.535385 2556 generic.go:356] "Generic (PLEG): container finished" podID="04d3e78f-c77a-4677-8b22-643f3e657964" containerID="b5117605dc571c49d593103055be547b2ac181ed804dd3b942cd1eda0ad81fd0" exitCode=0 Jul 09 21:56:49.535538 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:49.535408 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" event={"ID":"04d3e78f-c77a-4677-8b22-643f3e657964","Type":"ContainerDied","Data":"b5117605dc571c49d593103055be547b2ac181ed804dd3b942cd1eda0ad81fd0"} Jul 09 21:56:49.535756 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:49.535736 2556 scope.go:117] "RemoveContainer" containerID="b5117605dc571c49d593103055be547b2ac181ed804dd3b942cd1eda0ad81fd0" Jul 09 21:56:50.541165 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:50.541125 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-8q4rh" event={"ID":"04d3e78f-c77a-4677-8b22-643f3e657964","Type":"ContainerStarted","Data":"cde03c69ef53dc521596e9241c153ea40820bf36e4da07c5d4336a26ef0f4473"} Jul 09 21:56:50.542841 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:56:50.542821 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-6kzqk" event={"ID":"520a37a8-e07d-41c7-99c5-51bcbdd6ef23","Type":"ContainerStarted","Data":"9398024bb36839931272122f6153e5547460d390463c2059569ff169a6e28386"} Jul 09 21:57:12.890038 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:12.889959 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:57:12.890504 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:12.890233 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="alertmanager" containerID="cri-o://0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9" gracePeriod=120 Jul 09 21:57:12.890504 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:12.890276 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy-metric" containerID="cri-o://99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c" gracePeriod=120 Jul 09 21:57:12.890504 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:12.890311 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="prom-label-proxy" containerID="cri-o://65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280" gracePeriod=120 Jul 09 21:57:12.890504 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:12.890345 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="config-reloader" containerID="cri-o://0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560" gracePeriod=120 Jul 09 21:57:12.890504 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:12.890387 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy" containerID="cri-o://27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec" gracePeriod=120 Jul 09 21:57:12.890504 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:12.890344 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy-web" containerID="cri-o://64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b" gracePeriod=120 Jul 09 21:57:13.639825 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639793 2556 generic.go:356] "Generic (PLEG): container finished" podID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerID="65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280" exitCode=0 Jul 09 21:57:13.639825 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639819 2556 generic.go:356] "Generic (PLEG): container finished" podID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerID="99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c" exitCode=0 Jul 09 21:57:13.639825 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639825 2556 generic.go:356] "Generic (PLEG): container finished" podID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerID="27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec" exitCode=0 Jul 09 21:57:13.639825 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639830 2556 generic.go:356] "Generic (PLEG): container finished" podID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerID="0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560" exitCode=0 Jul 09 21:57:13.640084 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639836 2556 generic.go:356] "Generic (PLEG): container finished" podID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerID="0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9" exitCode=0 Jul 09 21:57:13.640084 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639861 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerDied","Data":"65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280"} Jul 09 21:57:13.640084 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639919 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerDied","Data":"99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c"} Jul 09 21:57:13.640084 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639944 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerDied","Data":"27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec"} Jul 09 21:57:13.640084 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639968 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerDied","Data":"0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560"} Jul 09 21:57:13.640084 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:13.639991 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerDied","Data":"0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9"} Jul 09 21:57:14.135209 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.135183 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.248749 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248663 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-web\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.248749 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248704 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-out\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.248749 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248732 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-cluster-tls-config\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248764 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-main-db\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248788 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-kube-api-access-4sh74\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248815 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248836 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248856 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-web-config\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248881 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-tls-assets\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248918 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248944 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249020 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.248968 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-metrics-client-ca\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249451 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.249030 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-volume\") pod \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\" (UID: \"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc\") " Jul 09 21:57:14.249451 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.249124 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 21:57:14.249451 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.249194 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 21:57:14.249451 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.249342 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.249451 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.249364 2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-alertmanager-main-db\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.250074 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.250047 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 21:57:14.251667 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.251624 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 21:57:14.251784 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.251764 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-out" (OuterVolumeSpecName: "config-out") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 21:57:14.252551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.252510 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 21:57:14.252551 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.252531 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-kube-api-access-4sh74" (OuterVolumeSpecName: "kube-api-access-4sh74") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "kube-api-access-4sh74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 21:57:14.252740 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.252697 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-volume" (OuterVolumeSpecName: "config-volume") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 21:57:14.252740 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.252723 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 21:57:14.253125 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.253098 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 21:57:14.253217 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.253183 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 21:57:14.256480 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.256371 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 21:57:14.262613 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.262594 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-web-config" (OuterVolumeSpecName: "web-config") pod "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" (UID: "58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 21:57:14.350440 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350403 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350455 2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-web-config\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350465 2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-tls-assets\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350473 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-main-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350482 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350491 2556 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-metrics-client-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350499 2556 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-volume\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350507 2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350515 2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-config-out\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350524 2556 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-cluster-tls-config\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.350617 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.350531 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sh74\" (UniqueName: \"kubernetes.io/projected/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc-kube-api-access-4sh74\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:57:14.646927 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.646893 2556 generic.go:356] "Generic (PLEG): container finished" podID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerID="64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b" exitCode=0 Jul 09 21:57:14.647110 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.646965 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerDied","Data":"64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b"} Jul 09 21:57:14.647110 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.646983 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.647110 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.647030 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc","Type":"ContainerDied","Data":"d32b335c5cfb675b11eb9d699142a6535de60be4c260dd7eb80bec2b35647bfc"} Jul 09 21:57:14.647110 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.647063 2556 scope.go:117] "RemoveContainer" containerID="65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280" Jul 09 21:57:14.655865 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.655769 2556 scope.go:117] "RemoveContainer" containerID="99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c" Jul 09 21:57:14.665033 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.665011 2556 scope.go:117] "RemoveContainer" containerID="27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec" Jul 09 21:57:14.671970 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.671948 2556 scope.go:117] "RemoveContainer" containerID="64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b" Jul 09 21:57:14.674478 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.674452 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:57:14.679179 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.679158 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:57:14.679593 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.679567 2556 scope.go:117] "RemoveContainer" containerID="0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560" Jul 09 21:57:14.686835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.686817 2556 scope.go:117] "RemoveContainer" containerID="0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9" Jul 09 21:57:14.693918 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.693901 2556 scope.go:117] "RemoveContainer" containerID="c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c" Jul 09 21:57:14.700814 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.700801 2556 scope.go:117] "RemoveContainer" containerID="65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280" Jul 09 21:57:14.701066 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:57:14.701047 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280\": container with ID starting with 65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280 not found: ID does not exist" containerID="65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280" Jul 09 21:57:14.701114 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.701077 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280"} err="failed to get container status \"65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280\": rpc error: code = NotFound desc = could not find container \"65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280\": container with ID starting with 65eb87f723e557fa0b17242f4684bf6703258407834a5a67bfdc4fd26a3ce280 not found: ID does not exist" Jul 09 21:57:14.701114 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.701109 2556 scope.go:117] "RemoveContainer" containerID="99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c" Jul 09 21:57:14.701385 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:57:14.701367 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c\": container with ID starting with 99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c not found: ID does not exist" containerID="99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c" Jul 09 21:57:14.701475 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.701390 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c"} err="failed to get container status \"99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c\": rpc error: code = NotFound desc = could not find container \"99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c\": container with ID starting with 99cbc938410cf22b77a821f7e0ca5c63f730bf571d5a8fe8583ac31618bcbf1c not found: ID does not exist" Jul 09 21:57:14.701475 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.701405 2556 scope.go:117] "RemoveContainer" containerID="27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec" Jul 09 21:57:14.701677 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:57:14.701661 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec\": container with ID starting with 27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec not found: ID does not exist" containerID="27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec" Jul 09 21:57:14.701721 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.701682 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec"} err="failed to get container status \"27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec\": rpc error: code = NotFound desc = could not find container \"27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec\": container with ID starting with 27f99f3e02aee032eb0e5030476dce617a98a82b73a02e239d37aed6602d8dec not found: ID does not exist" Jul 09 21:57:14.701721 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.701698 2556 scope.go:117] "RemoveContainer" containerID="64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b" Jul 09 21:57:14.701915 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:57:14.701899 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b\": container with ID starting with 64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b not found: ID does not exist" containerID="64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b" Jul 09 21:57:14.701955 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.701920 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b"} err="failed to get container status \"64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b\": rpc error: code = NotFound desc = could not find container \"64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b\": container with ID starting with 64017cd17f3c7c8fd99c727f38008f68fbe17ef28e086befce791aacc08ae31b not found: ID does not exist" Jul 09 21:57:14.701955 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.701937 2556 scope.go:117] "RemoveContainer" containerID="0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560" Jul 09 21:57:14.702133 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:57:14.702121 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560\": container with ID starting with 0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560 not found: ID does not exist" containerID="0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560" Jul 09 21:57:14.702176 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.702137 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560"} err="failed to get container status \"0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560\": rpc error: code = NotFound desc = could not find container \"0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560\": container with ID starting with 0b8004af524542b1fc1b17da4b87f36461e7a67e9d95eef474d0f48202f36560 not found: ID does not exist" Jul 09 21:57:14.702176 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.702148 2556 scope.go:117] "RemoveContainer" containerID="0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9" Jul 09 21:57:14.702338 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:57:14.702319 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9\": container with ID starting with 0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9 not found: ID does not exist" containerID="0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9" Jul 09 21:57:14.702381 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.702343 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9"} err="failed to get container status \"0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9\": rpc error: code = NotFound desc = could not find container \"0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9\": container with ID starting with 0c4e190cc02c40f9063adda8eacb1dabd990545a60a10419ff99f859beba94a9 not found: ID does not exist" Jul 09 21:57:14.702381 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.702357 2556 scope.go:117] "RemoveContainer" containerID="c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c" Jul 09 21:57:14.702547 ip-10-0-143-160 kubenswrapper[2556]: E0709 21:57:14.702533 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c\": container with ID starting with c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c not found: ID does not exist" containerID="c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c" Jul 09 21:57:14.702600 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.702552 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c"} err="failed to get container status \"c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c\": rpc error: code = NotFound desc = could not find container \"c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c\": container with ID starting with c864c9294ed812d5102025be04c32a9100040025469185c7ba21006a51a6eb1c not found: ID does not exist" Jul 09 21:57:14.705059 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705040 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:57:14.705317 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705307 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="alertmanager" Jul 09 21:57:14.705357 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705319 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="alertmanager" Jul 09 21:57:14.705357 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705331 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy-metric" Jul 09 21:57:14.705357 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705336 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy-metric" Jul 09 21:57:14.705357 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705345 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="config-reloader" Jul 09 21:57:14.705357 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705351 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="config-reloader" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705366 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="init-config-reloader" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705372 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="init-config-reloader" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705381 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705386 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705393 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy-web" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705399 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy-web" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705404 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="prom-label-proxy" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705409 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="prom-label-proxy" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705454 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="alertmanager" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705461 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705467 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="prom-label-proxy" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705472 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy-web" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705479 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="config-reloader" Jul 09 21:57:14.705503 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.705485 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" containerName="kube-rbac-proxy-metric" Jul 09 21:57:14.707679 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.707653 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.711309 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.711287 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 21:57:14.711418 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.711367 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 21:57:14.711488 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.711420 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 21:57:14.711488 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.711444 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kssn8\"" Jul 09 21:57:14.711796 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.711778 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 21:57:14.711796 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.711778 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 21:57:14.711948 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.711783 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 21:57:14.711948 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.711860 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 21:57:14.712085 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.712070 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 21:57:14.717278 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.717259 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 21:57:14.723545 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.723521 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:57:14.853247 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853201 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031b392d-83f4-493c-a7c4-1eb8af8e2b19-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853247 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853251 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853513 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853325 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htb5z\" (UniqueName: \"kubernetes.io/projected/031b392d-83f4-493c-a7c4-1eb8af8e2b19-kube-api-access-htb5z\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853513 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853379 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853513 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853406 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/031b392d-83f4-493c-a7c4-1eb8af8e2b19-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853513 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853444 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/031b392d-83f4-493c-a7c4-1eb8af8e2b19-config-out\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853513 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853467 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/031b392d-83f4-493c-a7c4-1eb8af8e2b19-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853513 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853508 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853836 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853556 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-config-volume\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853836 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853621 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/031b392d-83f4-493c-a7c4-1eb8af8e2b19-tls-assets\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853836 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853647 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853836 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853696 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.853836 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.853722 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-web-config\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954487 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954458 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031b392d-83f4-493c-a7c4-1eb8af8e2b19-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954495 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954516 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htb5z\" (UniqueName: \"kubernetes.io/projected/031b392d-83f4-493c-a7c4-1eb8af8e2b19-kube-api-access-htb5z\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954535 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954550 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/031b392d-83f4-493c-a7c4-1eb8af8e2b19-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954597 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954589 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/031b392d-83f4-493c-a7c4-1eb8af8e2b19-config-out\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954614 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/031b392d-83f4-493c-a7c4-1eb8af8e2b19-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954641 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-config-volume\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954679 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/031b392d-83f4-493c-a7c4-1eb8af8e2b19-tls-assets\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954693 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954725 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.954835 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.954741 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-web-config\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.955700 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.955610 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/031b392d-83f4-493c-a7c4-1eb8af8e2b19-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.955875 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.955852 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/031b392d-83f4-493c-a7c4-1eb8af8e2b19-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.957703 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.957681 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-web-config\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.958016 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.957838 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.958016 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.957847 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.958016 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.957894 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.958016 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.957969 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/031b392d-83f4-493c-a7c4-1eb8af8e2b19-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.958230 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.958083 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/031b392d-83f4-493c-a7c4-1eb8af8e2b19-config-out\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.958230 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.958136 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/031b392d-83f4-493c-a7c4-1eb8af8e2b19-tls-assets\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.958230 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.958150 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-config-volume\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.958393 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.958378 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.959710 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.959692 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/031b392d-83f4-493c-a7c4-1eb8af8e2b19-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:14.986470 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:14.986443 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htb5z\" (UniqueName: \"kubernetes.io/projected/031b392d-83f4-493c-a7c4-1eb8af8e2b19-kube-api-access-htb5z\") pod \"alertmanager-main-0\" (UID: \"031b392d-83f4-493c-a7c4-1eb8af8e2b19\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:15.018943 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:15.018921 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 21:57:15.158549 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:15.158526 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 21:57:15.160660 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:57:15.160624 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031b392d_83f4_493c_a7c4_1eb8af8e2b19.slice/crio-04756c3d53da9fdda16ac895b627a9733c16e1c68332bd2357602b2f2c094f2b WatchSource:0}: Error finding container 04756c3d53da9fdda16ac895b627a9733c16e1c68332bd2357602b2f2c094f2b: Status 404 returned error can't find the container with id 04756c3d53da9fdda16ac895b627a9733c16e1c68332bd2357602b2f2c094f2b Jul 09 21:57:15.274456 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:15.274426 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc" path="/var/lib/kubelet/pods/58ab1fda-f579-4078-b8fe-c9f5b5eeb4fc/volumes" Jul 09 21:57:15.652187 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:15.652156 2556 generic.go:356] "Generic (PLEG): container finished" podID="031b392d-83f4-493c-a7c4-1eb8af8e2b19" containerID="181fbab1c869b751b91e75c9a51fe1ff52db82c1f2d883fe89f88a0d56f7396f" exitCode=0 Jul 09 21:57:15.652366 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:15.652240 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"031b392d-83f4-493c-a7c4-1eb8af8e2b19","Type":"ContainerDied","Data":"181fbab1c869b751b91e75c9a51fe1ff52db82c1f2d883fe89f88a0d56f7396f"} Jul 09 21:57:15.652366 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:15.652300 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"031b392d-83f4-493c-a7c4-1eb8af8e2b19","Type":"ContainerStarted","Data":"04756c3d53da9fdda16ac895b627a9733c16e1c68332bd2357602b2f2c094f2b"} Jul 09 21:57:16.660398 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.660364 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"031b392d-83f4-493c-a7c4-1eb8af8e2b19","Type":"ContainerStarted","Data":"b2b491a8866bcfbba358ec1431d495d3ee710df14d927a47c3704685f4108146"} Jul 09 21:57:16.660808 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.660459 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"031b392d-83f4-493c-a7c4-1eb8af8e2b19","Type":"ContainerStarted","Data":"85e0c44988e6b1b5faf6b3c356a5bc1b321caae20868e2d7d44a81c6306c3177"} Jul 09 21:57:16.660808 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.660484 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"031b392d-83f4-493c-a7c4-1eb8af8e2b19","Type":"ContainerStarted","Data":"88a458817b71f60b5b0abe40a782dac1c86652806d625f9a327aa6cafe2ba00d"} Jul 09 21:57:16.660808 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.660509 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"031b392d-83f4-493c-a7c4-1eb8af8e2b19","Type":"ContainerStarted","Data":"4bc7550b40f303e06008ddea8989a1a406576d9da5d029dec54fa2275814147b"} Jul 09 21:57:16.660808 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.660531 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"031b392d-83f4-493c-a7c4-1eb8af8e2b19","Type":"ContainerStarted","Data":"66de2136bcc34d7af9941455063c9a336066859c5ddcd82bc122ecff03a7a602"} Jul 09 21:57:16.660808 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.660556 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"031b392d-83f4-493c-a7c4-1eb8af8e2b19","Type":"ContainerStarted","Data":"e526118746fda7a417824bb2f3f9b9e4abd35c6a432796819844d80b7bb80726"} Jul 09 21:57:16.694502 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.694456 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.67783668 podStartE2EDuration="2.694440661s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.603981ms" isStatefulPod=false podCreationTimestamp="2026-07-09 21:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:57:16.693451854 +0000 UTC m=+356.004069377" watchObservedRunningTime="2026-07-09 21:57:16.694440661 +0000 UTC m=+356.005058184" Jul 09 21:57:16.938967 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.938877 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst"] Jul 09 21:57:16.941902 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.941884 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:16.944742 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.944712 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 09 21:57:16.944969 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.944950 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 09 21:57:16.945294 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.945271 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 09 21:57:16.945406 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.945285 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 09 21:57:16.945406 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.945309 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-277j6\"" Jul 09 21:57:16.945504 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.945421 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 09 21:57:16.949417 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.949398 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 09 21:57:16.956320 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:16.956298 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst"] Jul 09 21:57:17.071962 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.071916 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-serving-certs-ca-bundle\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.071962 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.071958 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.072213 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.072010 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-telemeter-client-tls\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.072213 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.072052 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.072213 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.072086 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-metrics-client-ca\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.072213 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.072126 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-federate-client-tls\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.072213 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.072158 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-secret-telemeter-client\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.072213 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.072181 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz88c\" (UniqueName: \"kubernetes.io/projected/a48308d8-19a9-485a-9116-8726093d7174-kube-api-access-lz88c\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.173324 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.173294 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-telemeter-client-tls\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.173324 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.173329 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.173556 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.173355 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-metrics-client-ca\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.173556 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.173377 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-federate-client-tls\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.173556 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.173415 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-secret-telemeter-client\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.173556 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.173459 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz88c\" (UniqueName: \"kubernetes.io/projected/a48308d8-19a9-485a-9116-8726093d7174-kube-api-access-lz88c\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.173556 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.173522 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-serving-certs-ca-bundle\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.173556 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.173549 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.174266 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.174206 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-metrics-client-ca\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.174266 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.174254 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-serving-certs-ca-bundle\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.174410 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.174317 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a48308d8-19a9-485a-9116-8726093d7174-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.176541 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.176515 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-secret-telemeter-client\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.176646 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.176518 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-telemeter-client-tls\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.176646 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.176563 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-federate-client-tls\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.176961 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.176942 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a48308d8-19a9-485a-9116-8726093d7174-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.182299 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.182281 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz88c\" (UniqueName: \"kubernetes.io/projected/a48308d8-19a9-485a-9116-8726093d7174-kube-api-access-lz88c\") pod \"telemeter-client-5bcb67bb6c-dtdst\" (UID: \"a48308d8-19a9-485a-9116-8726093d7174\") " pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.254266 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.254187 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" Jul 09 21:57:17.414777 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.414742 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst"] Jul 09 21:57:17.418630 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:57:17.418605 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48308d8_19a9_485a_9116_8726093d7174.slice/crio-3ba27d3820776c778e9c81575460767876563d5d6165c55c397e332b4fff12b2 WatchSource:0}: Error finding container 3ba27d3820776c778e9c81575460767876563d5d6165c55c397e332b4fff12b2: Status 404 returned error can't find the container with id 3ba27d3820776c778e9c81575460767876563d5d6165c55c397e332b4fff12b2 Jul 09 21:57:17.420531 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.420515 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 21:57:17.665012 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:17.664979 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" event={"ID":"a48308d8-19a9-485a-9116-8726093d7174","Type":"ContainerStarted","Data":"3ba27d3820776c778e9c81575460767876563d5d6165c55c397e332b4fff12b2"} Jul 09 21:57:19.675184 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:19.675098 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" event={"ID":"a48308d8-19a9-485a-9116-8726093d7174","Type":"ContainerStarted","Data":"8f9a97e5e859aa8e14e7088eac1e5dbb136dc3051b56a498aa841cb6b363cc94"} Jul 09 21:57:19.675184 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:19.675158 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" event={"ID":"a48308d8-19a9-485a-9116-8726093d7174","Type":"ContainerStarted","Data":"5315f3332dbc4a51084f74b2e2df8c81f5b9cc0c160e4c082ca739bdb78fc7b9"} Jul 09 21:57:19.675184 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:19.675176 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" event={"ID":"a48308d8-19a9-485a-9116-8726093d7174","Type":"ContainerStarted","Data":"84d9e37c7d9d0fc1bd0db1dccc7ef27a6e9d6a21e55bb130f5b45e34d7516fef"} Jul 09 21:57:19.697733 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:57:19.697691 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bcb67bb6c-dtdst" podStartSLOduration=1.74354403 podStartE2EDuration="3.697676239s" totalImagesPullingTime="1.954132209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:57:19.696160352 +0000 UTC m=+359.006777876" watchObservedRunningTime="2026-07-09 21:57:19.697676239 +0000 UTC m=+359.008293761" Jul 09 21:59:24.412488 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.412457 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh"] Jul 09 21:59:24.415798 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.415781 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.418356 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.418337 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qd2gt\"" Jul 09 21:59:24.418541 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.418527 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 21:59:24.419234 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.419222 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 21:59:24.424717 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.424697 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh"] Jul 09 21:59:24.528141 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.528106 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqj47\" (UniqueName: \"kubernetes.io/projected/a48a9948-874b-40ba-9176-45234b127722-kube-api-access-vqj47\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.528311 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.528184 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.528311 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.528228 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.629142 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.629096 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.629272 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.629162 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqj47\" (UniqueName: \"kubernetes.io/projected/a48a9948-874b-40ba-9176-45234b127722-kube-api-access-vqj47\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.629272 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.629224 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.629502 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.629484 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.629549 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.629535 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.640399 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.640368 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqj47\" (UniqueName: \"kubernetes.io/projected/a48a9948-874b-40ba-9176-45234b127722-kube-api-access-vqj47\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.725633 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.725552 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:24.854517 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:24.854486 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh"] Jul 09 21:59:24.857525 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:59:24.857502 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48a9948_874b_40ba_9176_45234b127722.slice/crio-88dba2d6262a963afe988c1c700edcc700840b40098c8d56211bfaa51eda0a81 WatchSource:0}: Error finding container 88dba2d6262a963afe988c1c700edcc700840b40098c8d56211bfaa51eda0a81: Status 404 returned error can't find the container with id 88dba2d6262a963afe988c1c700edcc700840b40098c8d56211bfaa51eda0a81 Jul 09 21:59:25.162244 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:25.162199 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" event={"ID":"a48a9948-874b-40ba-9176-45234b127722","Type":"ContainerStarted","Data":"88dba2d6262a963afe988c1c700edcc700840b40098c8d56211bfaa51eda0a81"} Jul 09 21:59:32.193476 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:32.193439 2556 generic.go:356] "Generic (PLEG): container finished" podID="a48a9948-874b-40ba-9176-45234b127722" containerID="625c19de8422e89a7a88332aff15a44cf12a3b8f32b5060ee3c6083619019546" exitCode=0 Jul 09 21:59:32.193881 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:32.193511 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" event={"ID":"a48a9948-874b-40ba-9176-45234b127722","Type":"ContainerDied","Data":"625c19de8422e89a7a88332aff15a44cf12a3b8f32b5060ee3c6083619019546"} Jul 09 21:59:35.207773 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:35.207737 2556 generic.go:356] "Generic (PLEG): container finished" podID="a48a9948-874b-40ba-9176-45234b127722" containerID="84f555fb85c6a396aa4ab04cb20d9f719f9fdcadb9dfef0261ea0899c5ccc245" exitCode=0 Jul 09 21:59:35.208140 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:35.207814 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" event={"ID":"a48a9948-874b-40ba-9176-45234b127722","Type":"ContainerDied","Data":"84f555fb85c6a396aa4ab04cb20d9f719f9fdcadb9dfef0261ea0899c5ccc245"} Jul 09 21:59:44.250791 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:44.250756 2556 generic.go:356] "Generic (PLEG): container finished" podID="a48a9948-874b-40ba-9176-45234b127722" containerID="69c0f387c96718e2a72c0e81da55aa6500c914e8e7c07daafedad3334c3e45b6" exitCode=0 Jul 09 21:59:44.251186 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:44.250807 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" event={"ID":"a48a9948-874b-40ba-9176-45234b127722","Type":"ContainerDied","Data":"69c0f387c96718e2a72c0e81da55aa6500c914e8e7c07daafedad3334c3e45b6"} Jul 09 21:59:45.386869 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.386842 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:45.508686 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.508601 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqj47\" (UniqueName: \"kubernetes.io/projected/a48a9948-874b-40ba-9176-45234b127722-kube-api-access-vqj47\") pod \"a48a9948-874b-40ba-9176-45234b127722\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " Jul 09 21:59:45.508686 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.508664 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-util\") pod \"a48a9948-874b-40ba-9176-45234b127722\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " Jul 09 21:59:45.508906 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.508732 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-bundle\") pod \"a48a9948-874b-40ba-9176-45234b127722\" (UID: \"a48a9948-874b-40ba-9176-45234b127722\") " Jul 09 21:59:45.509331 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.509308 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-bundle" (OuterVolumeSpecName: "bundle") pod "a48a9948-874b-40ba-9176-45234b127722" (UID: "a48a9948-874b-40ba-9176-45234b127722"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 21:59:45.510775 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.510752 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48a9948-874b-40ba-9176-45234b127722-kube-api-access-vqj47" (OuterVolumeSpecName: "kube-api-access-vqj47") pod "a48a9948-874b-40ba-9176-45234b127722" (UID: "a48a9948-874b-40ba-9176-45234b127722"). InnerVolumeSpecName "kube-api-access-vqj47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 21:59:45.512501 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.512482 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-util" (OuterVolumeSpecName: "util") pod "a48a9948-874b-40ba-9176-45234b127722" (UID: "a48a9948-874b-40ba-9176-45234b127722"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 21:59:45.610167 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.610130 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqj47\" (UniqueName: \"kubernetes.io/projected/a48a9948-874b-40ba-9176-45234b127722-kube-api-access-vqj47\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:59:45.610167 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.610159 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-util\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:59:45.610167 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:45.610170 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a48a9948-874b-40ba-9176-45234b127722-bundle\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 21:59:46.261177 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:46.261139 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" event={"ID":"a48a9948-874b-40ba-9176-45234b127722","Type":"ContainerDied","Data":"88dba2d6262a963afe988c1c700edcc700840b40098c8d56211bfaa51eda0a81"} Jul 09 21:59:46.261177 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:46.261169 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404zzjh" Jul 09 21:59:46.261373 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:46.261196 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88dba2d6262a963afe988c1c700edcc700840b40098c8d56211bfaa51eda0a81" Jul 09 21:59:52.415535 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.415498 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb"] Jul 09 21:59:52.416121 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.415990 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a48a9948-874b-40ba-9176-45234b127722" containerName="pull" Jul 09 21:59:52.416121 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.416008 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a48a9948-874b-40ba-9176-45234b127722" containerName="pull" Jul 09 21:59:52.416121 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.416034 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a48a9948-874b-40ba-9176-45234b127722" containerName="extract" Jul 09 21:59:52.416121 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.416043 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a48a9948-874b-40ba-9176-45234b127722" containerName="extract" Jul 09 21:59:52.416121 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.416067 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a48a9948-874b-40ba-9176-45234b127722" containerName="util" Jul 09 21:59:52.416121 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.416076 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a48a9948-874b-40ba-9176-45234b127722" containerName="util" Jul 09 21:59:52.416492 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.416154 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a48a9948-874b-40ba-9176-45234b127722" containerName="extract" Jul 09 21:59:52.419911 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.419890 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" Jul 09 21:59:52.422430 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.422409 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 09 21:59:52.422564 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.422546 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 09 21:59:52.422663 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.422548 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ksj74\"" Jul 09 21:59:52.431138 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.431116 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb"] Jul 09 21:59:52.567666 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.567623 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxrl2\" (UniqueName: \"kubernetes.io/projected/79366fa8-4e4c-4824-a87d-33534f2e08dd-kube-api-access-xxrl2\") pod \"cert-manager-operator-controller-manager-55c444bf86-2n8xb\" (UID: \"79366fa8-4e4c-4824-a87d-33534f2e08dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" Jul 09 21:59:52.567666 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.567669 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79366fa8-4e4c-4824-a87d-33534f2e08dd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2n8xb\" (UID: \"79366fa8-4e4c-4824-a87d-33534f2e08dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" Jul 09 21:59:52.668765 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.668679 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxrl2\" (UniqueName: \"kubernetes.io/projected/79366fa8-4e4c-4824-a87d-33534f2e08dd-kube-api-access-xxrl2\") pod \"cert-manager-operator-controller-manager-55c444bf86-2n8xb\" (UID: \"79366fa8-4e4c-4824-a87d-33534f2e08dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" Jul 09 21:59:52.668765 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.668719 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79366fa8-4e4c-4824-a87d-33534f2e08dd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2n8xb\" (UID: \"79366fa8-4e4c-4824-a87d-33534f2e08dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" Jul 09 21:59:52.669069 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.669053 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79366fa8-4e4c-4824-a87d-33534f2e08dd-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-2n8xb\" (UID: \"79366fa8-4e4c-4824-a87d-33534f2e08dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" Jul 09 21:59:52.681470 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.681439 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxrl2\" (UniqueName: \"kubernetes.io/projected/79366fa8-4e4c-4824-a87d-33534f2e08dd-kube-api-access-xxrl2\") pod \"cert-manager-operator-controller-manager-55c444bf86-2n8xb\" (UID: \"79366fa8-4e4c-4824-a87d-33534f2e08dd\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" Jul 09 21:59:52.729548 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.729512 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" Jul 09 21:59:52.860084 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:52.860061 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb"] Jul 09 21:59:52.862537 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:59:52.862513 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79366fa8_4e4c_4824_a87d_33534f2e08dd.slice/crio-0bf825435dde31440e2244ee30925aa1fbb9bcd1b11f957a48f7d59521bb2ecd WatchSource:0}: Error finding container 0bf825435dde31440e2244ee30925aa1fbb9bcd1b11f957a48f7d59521bb2ecd: Status 404 returned error can't find the container with id 0bf825435dde31440e2244ee30925aa1fbb9bcd1b11f957a48f7d59521bb2ecd Jul 09 21:59:53.290243 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:53.290205 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" event={"ID":"79366fa8-4e4c-4824-a87d-33534f2e08dd","Type":"ContainerStarted","Data":"0bf825435dde31440e2244ee30925aa1fbb9bcd1b11f957a48f7d59521bb2ecd"} Jul 09 21:59:56.307146 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:56.307110 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" event={"ID":"79366fa8-4e4c-4824-a87d-33534f2e08dd","Type":"ContainerStarted","Data":"7f8b34f4c6390d3e74db2fde1bc1def7c1c17c61fb54df25fd256ea5f52507ab"} Jul 09 21:59:56.327194 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:56.327150 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-2n8xb" podStartSLOduration=1.814832577 podStartE2EDuration="4.327136287s" totalImagesPullingTime="2.51230371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 21:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 21:59:56.325272575 +0000 UTC m=+515.635890110" watchObservedRunningTime="2026-07-09 21:59:56.327136287 +0000 UTC m=+515.637753810" Jul 09 21:59:56.923750 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:56.923716 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l"] Jul 09 21:59:56.926421 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:56.926407 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:56.928971 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:56.928948 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 21:59:56.929095 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:56.928996 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qd2gt\"" Jul 09 21:59:56.929859 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:56.929845 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 21:59:56.935912 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:56.935886 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l"] Jul 09 21:59:57.007883 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.007854 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7pcv\" (UniqueName: \"kubernetes.io/projected/e6fb1dcf-2649-4448-9b10-f9716ffc9951-kube-api-access-g7pcv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.008024 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.007897 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.008024 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.007962 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.109047 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.109003 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7pcv\" (UniqueName: \"kubernetes.io/projected/e6fb1dcf-2649-4448-9b10-f9716ffc9951-kube-api-access-g7pcv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.109228 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.109066 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.109228 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.109117 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.109458 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.109435 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.109525 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.109475 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.117143 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.117119 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7pcv\" (UniqueName: \"kubernetes.io/projected/e6fb1dcf-2649-4448-9b10-f9716ffc9951-kube-api-access-g7pcv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.236498 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.236403 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 21:59:57.363398 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:57.363255 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l"] Jul 09 21:59:57.365469 ip-10-0-143-160 kubenswrapper[2556]: W0709 21:59:57.365443 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fb1dcf_2649_4448_9b10_f9716ffc9951.slice/crio-ee9994156e57339e61cdb9b2d94b5455a40cec5292d016ce5d8d701a5e0097b5 WatchSource:0}: Error finding container ee9994156e57339e61cdb9b2d94b5455a40cec5292d016ce5d8d701a5e0097b5: Status 404 returned error can't find the container with id ee9994156e57339e61cdb9b2d94b5455a40cec5292d016ce5d8d701a5e0097b5 Jul 09 21:59:58.317725 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:58.317689 2556 generic.go:356] "Generic (PLEG): container finished" podID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerID="8a0d0608823f46d5a773c77dacc45c137a73a5eff0cf8717a3ad62452c45f825" exitCode=0 Jul 09 21:59:58.317894 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:58.317772 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" event={"ID":"e6fb1dcf-2649-4448-9b10-f9716ffc9951","Type":"ContainerDied","Data":"8a0d0608823f46d5a773c77dacc45c137a73a5eff0cf8717a3ad62452c45f825"} Jul 09 21:59:58.317894 ip-10-0-143-160 kubenswrapper[2556]: I0709 21:59:58.317824 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" event={"ID":"e6fb1dcf-2649-4448-9b10-f9716ffc9951","Type":"ContainerStarted","Data":"ee9994156e57339e61cdb9b2d94b5455a40cec5292d016ce5d8d701a5e0097b5"} Jul 09 22:00:07.147310 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.147274 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vhg2k"] Jul 09 22:00:07.149841 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.149824 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vhg2k" Jul 09 22:00:07.152219 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.152195 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 22:00:07.153110 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.153090 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-f2mvg\"" Jul 09 22:00:07.153192 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.153138 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 22:00:07.161173 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.161149 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vhg2k"] Jul 09 22:00:07.194900 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.194861 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9d67406-08d4-4509-bf13-310b4eb8404f-bound-sa-token\") pod \"cert-manager-55b4d954d5-vhg2k\" (UID: \"a9d67406-08d4-4509-bf13-310b4eb8404f\") " pod="cert-manager/cert-manager-55b4d954d5-vhg2k" Jul 09 22:00:07.194900 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.194896 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmmwl\" (UniqueName: \"kubernetes.io/projected/a9d67406-08d4-4509-bf13-310b4eb8404f-kube-api-access-lmmwl\") pod \"cert-manager-55b4d954d5-vhg2k\" (UID: \"a9d67406-08d4-4509-bf13-310b4eb8404f\") " pod="cert-manager/cert-manager-55b4d954d5-vhg2k" Jul 09 22:00:07.296095 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.296052 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9d67406-08d4-4509-bf13-310b4eb8404f-bound-sa-token\") pod \"cert-manager-55b4d954d5-vhg2k\" (UID: \"a9d67406-08d4-4509-bf13-310b4eb8404f\") " pod="cert-manager/cert-manager-55b4d954d5-vhg2k" Jul 09 22:00:07.296269 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.296111 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmmwl\" (UniqueName: \"kubernetes.io/projected/a9d67406-08d4-4509-bf13-310b4eb8404f-kube-api-access-lmmwl\") pod \"cert-manager-55b4d954d5-vhg2k\" (UID: \"a9d67406-08d4-4509-bf13-310b4eb8404f\") " pod="cert-manager/cert-manager-55b4d954d5-vhg2k" Jul 09 22:00:07.305185 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.305154 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a9d67406-08d4-4509-bf13-310b4eb8404f-bound-sa-token\") pod \"cert-manager-55b4d954d5-vhg2k\" (UID: \"a9d67406-08d4-4509-bf13-310b4eb8404f\") " pod="cert-manager/cert-manager-55b4d954d5-vhg2k" Jul 09 22:00:07.305323 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.305309 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmmwl\" (UniqueName: \"kubernetes.io/projected/a9d67406-08d4-4509-bf13-310b4eb8404f-kube-api-access-lmmwl\") pod \"cert-manager-55b4d954d5-vhg2k\" (UID: \"a9d67406-08d4-4509-bf13-310b4eb8404f\") " pod="cert-manager/cert-manager-55b4d954d5-vhg2k" Jul 09 22:00:07.473451 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.473357 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vhg2k" Jul 09 22:00:07.604983 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:07.604947 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vhg2k"] Jul 09 22:00:07.608640 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:00:07.608608 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9d67406_08d4_4509_bf13_310b4eb8404f.slice/crio-f8b6a8e47f0d884b145264e676cea3a57a8c5faee187cc33e9f17c39bb7876cf WatchSource:0}: Error finding container f8b6a8e47f0d884b145264e676cea3a57a8c5faee187cc33e9f17c39bb7876cf: Status 404 returned error can't find the container with id f8b6a8e47f0d884b145264e676cea3a57a8c5faee187cc33e9f17c39bb7876cf Jul 09 22:00:08.369997 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:08.369960 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vhg2k" event={"ID":"a9d67406-08d4-4509-bf13-310b4eb8404f","Type":"ContainerStarted","Data":"f8b6a8e47f0d884b145264e676cea3a57a8c5faee187cc33e9f17c39bb7876cf"} Jul 09 22:00:09.376181 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:09.376146 2556 generic.go:356] "Generic (PLEG): container finished" podID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerID="a0ea58c7ce1b92004c2d4c616fe4e73b9164d2578faaca370c41dbc470564c75" exitCode=0 Jul 09 22:00:09.376552 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:09.376196 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" event={"ID":"e6fb1dcf-2649-4448-9b10-f9716ffc9951","Type":"ContainerDied","Data":"a0ea58c7ce1b92004c2d4c616fe4e73b9164d2578faaca370c41dbc470564c75"} Jul 09 22:00:10.382865 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:10.382832 2556 generic.go:356] "Generic (PLEG): container finished" podID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerID="62bd75fbc9192675b97459a697812e09bae078824d52d9c41747cae2590199a8" exitCode=0 Jul 09 22:00:10.383243 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:10.382889 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" event={"ID":"e6fb1dcf-2649-4448-9b10-f9716ffc9951","Type":"ContainerDied","Data":"62bd75fbc9192675b97459a697812e09bae078824d52d9c41747cae2590199a8"} Jul 09 22:00:11.551603 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.551559 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 22:00:11.632241 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.632205 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7pcv\" (UniqueName: \"kubernetes.io/projected/e6fb1dcf-2649-4448-9b10-f9716ffc9951-kube-api-access-g7pcv\") pod \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " Jul 09 22:00:11.632414 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.632335 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-bundle\") pod \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " Jul 09 22:00:11.632414 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.632390 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-util\") pod \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\" (UID: \"e6fb1dcf-2649-4448-9b10-f9716ffc9951\") " Jul 09 22:00:11.632816 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.632776 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-bundle" (OuterVolumeSpecName: "bundle") pod "e6fb1dcf-2649-4448-9b10-f9716ffc9951" (UID: "e6fb1dcf-2649-4448-9b10-f9716ffc9951"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:00:11.634568 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.634542 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fb1dcf-2649-4448-9b10-f9716ffc9951-kube-api-access-g7pcv" (OuterVolumeSpecName: "kube-api-access-g7pcv") pod "e6fb1dcf-2649-4448-9b10-f9716ffc9951" (UID: "e6fb1dcf-2649-4448-9b10-f9716ffc9951"). InnerVolumeSpecName "kube-api-access-g7pcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:00:11.637271 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.637248 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-util" (OuterVolumeSpecName: "util") pod "e6fb1dcf-2649-4448-9b10-f9716ffc9951" (UID: "e6fb1dcf-2649-4448-9b10-f9716ffc9951"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:00:11.733434 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.733401 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-bundle\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:00:11.733434 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.733431 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6fb1dcf-2649-4448-9b10-f9716ffc9951-util\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:00:11.733684 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:11.733445 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7pcv\" (UniqueName: \"kubernetes.io/projected/e6fb1dcf-2649-4448-9b10-f9716ffc9951-kube-api-access-g7pcv\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:00:12.397066 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:12.396980 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" event={"ID":"e6fb1dcf-2649-4448-9b10-f9716ffc9951","Type":"ContainerDied","Data":"ee9994156e57339e61cdb9b2d94b5455a40cec5292d016ce5d8d701a5e0097b5"} Jul 09 22:00:12.397066 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:12.397018 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqk26l" Jul 09 22:00:12.397066 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:12.397035 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9994156e57339e61cdb9b2d94b5455a40cec5292d016ce5d8d701a5e0097b5" Jul 09 22:00:13.403359 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:13.403264 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vhg2k" event={"ID":"a9d67406-08d4-4509-bf13-310b4eb8404f","Type":"ContainerStarted","Data":"f76ed7bc151acc6d60b95bc36459195142937b28e14cec6c469ec085f2b655cf"} Jul 09 22:00:13.421908 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:13.421857 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vhg2k" podStartSLOduration=0.925590951 podStartE2EDuration="6.421840974s" totalImagesPullingTime="5.496250023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:00:13.420568643 +0000 UTC m=+532.731186162" watchObservedRunningTime="2026-07-09 22:00:13.421840974 +0000 UTC m=+532.732458496" Jul 09 22:00:27.120800 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.120764 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652"] Jul 09 22:00:27.121245 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.121146 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerName="util" Jul 09 22:00:27.121245 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.121161 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerName="util" Jul 09 22:00:27.121245 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.121178 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerName="extract" Jul 09 22:00:27.121245 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.121183 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerName="extract" Jul 09 22:00:27.121245 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.121198 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerName="pull" Jul 09 22:00:27.121245 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.121203 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerName="pull" Jul 09 22:00:27.121430 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.121264 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fb1dcf-2649-4448-9b10-f9716ffc9951" containerName="extract" Jul 09 22:00:27.123477 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.123461 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.125973 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.125949 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 22:00:27.125973 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.125961 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 22:00:27.126160 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.125950 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qd2gt\"" Jul 09 22:00:27.132938 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.132917 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652"] Jul 09 22:00:27.161422 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.161398 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.161538 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.161489 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.161538 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.161510 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctq28\" (UniqueName: \"kubernetes.io/projected/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-kube-api-access-ctq28\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.262442 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.262404 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.262442 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.262440 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctq28\" (UniqueName: \"kubernetes.io/projected/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-kube-api-access-ctq28\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.262662 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.262503 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.262969 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.262954 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.263005 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.262983 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.271164 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.271140 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctq28\" (UniqueName: \"kubernetes.io/projected/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-kube-api-access-ctq28\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.433118 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.433032 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:27.563652 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:27.563628 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652"] Jul 09 22:00:27.566023 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:00:27.565995 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2e4a3c_09ce_4014_ac3e_c17cb2cb1f03.slice/crio-9039e94136e50cef26e365e26a72ffc8e7e0695021a0f25727aea898c11fb421 WatchSource:0}: Error finding container 9039e94136e50cef26e365e26a72ffc8e7e0695021a0f25727aea898c11fb421: Status 404 returned error can't find the container with id 9039e94136e50cef26e365e26a72ffc8e7e0695021a0f25727aea898c11fb421 Jul 09 22:00:28.474904 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:28.474868 2556 generic.go:356] "Generic (PLEG): container finished" podID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerID="601d8cd8125e1964f88eeff02b2dbaf878e9c8874d5b024adcef2c34789232a9" exitCode=0 Jul 09 22:00:28.475281 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:28.474924 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" event={"ID":"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03","Type":"ContainerDied","Data":"601d8cd8125e1964f88eeff02b2dbaf878e9c8874d5b024adcef2c34789232a9"} Jul 09 22:00:28.475281 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:28.474960 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" event={"ID":"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03","Type":"ContainerStarted","Data":"9039e94136e50cef26e365e26a72ffc8e7e0695021a0f25727aea898c11fb421"} Jul 09 22:00:29.481275 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:29.481233 2556 generic.go:356] "Generic (PLEG): container finished" podID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerID="527ee8cca14775cf4b5f6e8cd5e9ce7ad3ba5ef28055b8d64d38925f560cad24" exitCode=0 Jul 09 22:00:29.481691 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:29.481289 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" event={"ID":"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03","Type":"ContainerDied","Data":"527ee8cca14775cf4b5f6e8cd5e9ce7ad3ba5ef28055b8d64d38925f560cad24"} Jul 09 22:00:30.488778 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:30.488744 2556 generic.go:356] "Generic (PLEG): container finished" podID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerID="55f57635abdd10743ad3b3c71eff93e84dfc40bbf6325972cbe22a3f34fdcfa5" exitCode=0 Jul 09 22:00:30.489156 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:30.488798 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" event={"ID":"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03","Type":"ContainerDied","Data":"55f57635abdd10743ad3b3c71eff93e84dfc40bbf6325972cbe22a3f34fdcfa5"} Jul 09 22:00:31.621498 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.621469 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:31.699899 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.699867 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-bundle\") pod \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " Jul 09 22:00:31.699899 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.699902 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctq28\" (UniqueName: \"kubernetes.io/projected/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-kube-api-access-ctq28\") pod \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " Jul 09 22:00:31.700159 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.699958 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-util\") pod \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\" (UID: \"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03\") " Jul 09 22:00:31.700692 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.700666 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-bundle" (OuterVolumeSpecName: "bundle") pod "3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" (UID: "3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:00:31.702028 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.702002 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-kube-api-access-ctq28" (OuterVolumeSpecName: "kube-api-access-ctq28") pod "3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" (UID: "3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03"). InnerVolumeSpecName "kube-api-access-ctq28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:00:31.705380 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.705342 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-util" (OuterVolumeSpecName: "util") pod "3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" (UID: "3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:00:31.800965 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.800889 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-util\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:00:31.800965 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.800915 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-bundle\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:00:31.800965 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:31.800925 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctq28\" (UniqueName: \"kubernetes.io/projected/3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03-kube-api-access-ctq28\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:00:32.500294 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:32.500265 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" Jul 09 22:00:32.500500 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:32.500294 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5g652" event={"ID":"3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03","Type":"ContainerDied","Data":"9039e94136e50cef26e365e26a72ffc8e7e0695021a0f25727aea898c11fb421"} Jul 09 22:00:32.500500 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:32.500340 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9039e94136e50cef26e365e26a72ffc8e7e0695021a0f25727aea898c11fb421" Jul 09 22:00:49.110141 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.107537 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f"] Jul 09 22:00:49.110141 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.108349 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerName="util" Jul 09 22:00:49.110141 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.108376 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerName="util" Jul 09 22:00:49.110141 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.108394 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerName="pull" Jul 09 22:00:49.110141 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.108402 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerName="pull" Jul 09 22:00:49.110141 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.108429 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerName="extract" Jul 09 22:00:49.110141 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.108437 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerName="extract" Jul 09 22:00:49.110141 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.108625 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2e4a3c-09ce-4014-ac3e-c17cb2cb1f03" containerName="extract" Jul 09 22:00:49.111930 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.111908 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.114871 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.114849 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 22:00:49.115832 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.115812 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 22:00:49.115832 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.115830 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 22:00:49.116004 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.115874 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zkh2d\"" Jul 09 22:00:49.116004 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.115944 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 22:00:49.116105 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.116041 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:00:49.124042 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.124021 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f"] Jul 09 22:00:49.251992 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.251947 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eeb51403-af15-4663-9fb6-d86d137c196e-manager-config\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.251992 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.252005 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ss7\" (UniqueName: \"kubernetes.io/projected/eeb51403-af15-4663-9fb6-d86d137c196e-kube-api-access-m4ss7\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.252208 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.252076 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeb51403-af15-4663-9fb6-d86d137c196e-cert\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.252208 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.252129 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb51403-af15-4663-9fb6-d86d137c196e-metrics-cert\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.352761 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.352727 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ss7\" (UniqueName: \"kubernetes.io/projected/eeb51403-af15-4663-9fb6-d86d137c196e-kube-api-access-m4ss7\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.352921 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.352773 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeb51403-af15-4663-9fb6-d86d137c196e-cert\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.352983 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.352957 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb51403-af15-4663-9fb6-d86d137c196e-metrics-cert\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.353078 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.353053 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eeb51403-af15-4663-9fb6-d86d137c196e-manager-config\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.353662 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.353643 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eeb51403-af15-4663-9fb6-d86d137c196e-manager-config\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.355228 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.355207 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeb51403-af15-4663-9fb6-d86d137c196e-cert\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.355323 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.355273 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eeb51403-af15-4663-9fb6-d86d137c196e-metrics-cert\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.361292 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.361228 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ss7\" (UniqueName: \"kubernetes.io/projected/eeb51403-af15-4663-9fb6-d86d137c196e-kube-api-access-m4ss7\") pod \"lws-controller-manager-7b9cdbd59d-f5c8f\" (UID: \"eeb51403-af15-4663-9fb6-d86d137c196e\") " pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.421689 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.421642 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:49.556117 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.556091 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f"] Jul 09 22:00:49.558555 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:00:49.558523 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb51403_af15_4663_9fb6_d86d137c196e.slice/crio-7c0bb0a6c8cb67e22eb075d7b719ac3b6fc4c7635df5e32888eb7d0b784849dc WatchSource:0}: Error finding container 7c0bb0a6c8cb67e22eb075d7b719ac3b6fc4c7635df5e32888eb7d0b784849dc: Status 404 returned error can't find the container with id 7c0bb0a6c8cb67e22eb075d7b719ac3b6fc4c7635df5e32888eb7d0b784849dc Jul 09 22:00:49.580208 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:49.580163 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" event={"ID":"eeb51403-af15-4663-9fb6-d86d137c196e","Type":"ContainerStarted","Data":"7c0bb0a6c8cb67e22eb075d7b719ac3b6fc4c7635df5e32888eb7d0b784849dc"} Jul 09 22:00:51.592121 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.592088 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" event={"ID":"eeb51403-af15-4663-9fb6-d86d137c196e","Type":"ContainerStarted","Data":"c91b45ad016cc417b2cd463fbf25dfe095a4dfdbc1a2e5c5e71b88f83c9fb53b"} Jul 09 22:00:51.592525 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.592223 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:00:51.613822 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.613771 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" podStartSLOduration=0.805470602 podStartE2EDuration="2.613754011s" totalImagesPullingTime="1.808283409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:00:51.612255905 +0000 UTC m=+570.922873428" watchObservedRunningTime="2026-07-09 22:00:51.613754011 +0000 UTC m=+570.924371534" Jul 09 22:00:51.799079 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.798996 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf"] Jul 09 22:00:51.801563 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.801549 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:51.804664 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.804646 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 22:00:51.805084 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.805062 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 22:00:51.805256 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.805238 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xz842\"" Jul 09 22:00:51.805332 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.805272 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 22:00:51.805332 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.805310 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 22:00:51.825976 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.825951 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf"] Jul 09 22:00:51.974000 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.973962 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de54257a-58b6-4c6b-989c-4bc460a32922-tmp\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:51.974179 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.974024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzzl6\" (UniqueName: \"kubernetes.io/projected/de54257a-58b6-4c6b-989c-4bc460a32922-kube-api-access-fzzl6\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:51.974179 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.974068 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de54257a-58b6-4c6b-989c-4bc460a32922-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:51.974179 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.974085 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de54257a-58b6-4c6b-989c-4bc460a32922-webhook-cert\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:51.974179 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:51.974124 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de54257a-58b6-4c6b-989c-4bc460a32922-manifests\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.075134 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.075043 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzzl6\" (UniqueName: \"kubernetes.io/projected/de54257a-58b6-4c6b-989c-4bc460a32922-kube-api-access-fzzl6\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.075134 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.075086 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de54257a-58b6-4c6b-989c-4bc460a32922-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.075134 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.075106 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de54257a-58b6-4c6b-989c-4bc460a32922-webhook-cert\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.075134 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.075136 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de54257a-58b6-4c6b-989c-4bc460a32922-manifests\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.075451 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.075300 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de54257a-58b6-4c6b-989c-4bc460a32922-tmp\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.075669 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.075642 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/de54257a-58b6-4c6b-989c-4bc460a32922-manifests\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.077373 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.077353 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de54257a-58b6-4c6b-989c-4bc460a32922-tmp\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.077603 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.077565 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/de54257a-58b6-4c6b-989c-4bc460a32922-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.077749 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.077730 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/de54257a-58b6-4c6b-989c-4bc460a32922-webhook-cert\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.082864 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.082843 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzzl6\" (UniqueName: \"kubernetes.io/projected/de54257a-58b6-4c6b-989c-4bc460a32922-kube-api-access-fzzl6\") pod \"opendatahub-operator-controller-manager-85dfbddc47-dsbqf\" (UID: \"de54257a-58b6-4c6b-989c-4bc460a32922\") " pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.111862 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.111841 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:52.250703 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.250674 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf"] Jul 09 22:00:52.252648 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:00:52.252623 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde54257a_58b6_4c6b_989c_4bc460a32922.slice/crio-8ec8c7981a3929d04731690147cd65bc80d7b1f45f19956276795bdc1dba9457 WatchSource:0}: Error finding container 8ec8c7981a3929d04731690147cd65bc80d7b1f45f19956276795bdc1dba9457: Status 404 returned error can't find the container with id 8ec8c7981a3929d04731690147cd65bc80d7b1f45f19956276795bdc1dba9457 Jul 09 22:00:52.598201 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:52.598169 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" event={"ID":"de54257a-58b6-4c6b-989c-4bc460a32922","Type":"ContainerStarted","Data":"8ec8c7981a3929d04731690147cd65bc80d7b1f45f19956276795bdc1dba9457"} Jul 09 22:00:56.620867 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:56.620829 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" event={"ID":"de54257a-58b6-4c6b-989c-4bc460a32922","Type":"ContainerStarted","Data":"f3609db1e84a0a1ce7554e6fefd61fe2fda635a5c7676abd76aed5d3916fee58"} Jul 09 22:00:57.626656 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:57.626621 2556 generic.go:356] "Generic (PLEG): container finished" podID="de54257a-58b6-4c6b-989c-4bc460a32922" containerID="f3609db1e84a0a1ce7554e6fefd61fe2fda635a5c7676abd76aed5d3916fee58" exitCode=0 Jul 09 22:00:57.627170 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:57.626711 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" event={"ID":"de54257a-58b6-4c6b-989c-4bc460a32922","Type":"ContainerDied","Data":"f3609db1e84a0a1ce7554e6fefd61fe2fda635a5c7676abd76aed5d3916fee58"} Jul 09 22:00:58.635175 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:58.635136 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" event={"ID":"de54257a-58b6-4c6b-989c-4bc460a32922","Type":"ContainerStarted","Data":"2a2d8c78ef21e6af1d7e0f843db2abeb65d07905cc4c9199ce4ffc6de4bb00f8"} Jul 09 22:00:58.635550 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:58.635285 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:00:58.659316 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:00:58.659249 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" podStartSLOduration=2.179268506 podStartE2EDuration="7.659230566s" totalImagesPullingTime="4.404473125s" totalInitContainerRuntime="1.075488935s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:00:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:00:58.656234694 +0000 UTC m=+577.966852217" watchObservedRunningTime="2026-07-09 22:00:58.659230566 +0000 UTC m=+577.969848090" Jul 09 22:01:02.600490 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:02.600459 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7b9cdbd59d-f5c8f" Jul 09 22:01:09.642164 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:09.642132 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85dfbddc47-dsbqf" Jul 09 22:01:21.134139 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:21.134115 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:01:21.134692 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:21.134156 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:01:29.298532 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.298500 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx"] Jul 09 22:01:29.306023 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.306002 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.308690 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.308671 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 22:01:29.308690 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.308683 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dwp7s\"" Jul 09 22:01:29.314082 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.314058 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx"] Jul 09 22:01:29.406718 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406681 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.406914 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406735 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14fc4f3d-f43e-4d5c-839c-902301586b07-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.406914 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406757 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.406914 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406782 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.406914 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406802 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.406914 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406858 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.406914 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406878 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9f9\" (UniqueName: \"kubernetes.io/projected/14fc4f3d-f43e-4d5c-839c-902301586b07-kube-api-access-6m9f9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.406914 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406902 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.407152 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.406933 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508353 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508318 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9f9\" (UniqueName: \"kubernetes.io/projected/14fc4f3d-f43e-4d5c-839c-902301586b07-kube-api-access-6m9f9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508551 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508360 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508551 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508399 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508551 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508431 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508551 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508475 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14fc4f3d-f43e-4d5c-839c-902301586b07-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508551 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508508 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508865 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508633 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508865 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508680 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508865 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508737 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.508865 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508845 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.509043 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.508908 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.509079 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.509065 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.509146 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.509120 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.509218 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.509201 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14fc4f3d-f43e-4d5c-839c-902301586b07-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.510883 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.510864 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.511132 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.511111 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.516776 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.516749 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9f9\" (UniqueName: \"kubernetes.io/projected/14fc4f3d-f43e-4d5c-839c-902301586b07-kube-api-access-6m9f9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.516776 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.516763 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14fc4f3d-f43e-4d5c-839c-902301586b07-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf46nkvx\" (UID: \"14fc4f3d-f43e-4d5c-839c-902301586b07\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.616878 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.616841 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:29.777892 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:29.777864 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx"] Jul 09 22:01:29.779965 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:01:29.779930 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fc4f3d_f43e_4d5c_839c_902301586b07.slice/crio-cd301e8cb4fc52b3c273d113eb5b525538fb289551ee25eb9b85d331c427bbc9 WatchSource:0}: Error finding container cd301e8cb4fc52b3c273d113eb5b525538fb289551ee25eb9b85d331c427bbc9: Status 404 returned error can't find the container with id cd301e8cb4fc52b3c273d113eb5b525538fb289551ee25eb9b85d331c427bbc9 Jul 09 22:01:30.786566 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:30.786522 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" event={"ID":"14fc4f3d-f43e-4d5c-839c-902301586b07","Type":"ContainerStarted","Data":"cd301e8cb4fc52b3c273d113eb5b525538fb289551ee25eb9b85d331c427bbc9"} Jul 09 22:01:31.220696 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.220664 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn"] Jul 09 22:01:31.224004 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.223980 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.227004 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.226956 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 22:01:31.227004 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.226973 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 22:01:31.227551 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.227030 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7cp4j\"" Jul 09 22:01:31.227551 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.227267 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 22:01:31.236183 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.236159 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn"] Jul 09 22:01:31.328684 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.328649 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.328867 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.328733 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9mr\" (UniqueName: \"kubernetes.io/projected/68ddcb3f-a771-48a3-bc49-75c332ea3532-kube-api-access-rx9mr\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.328867 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.328774 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ddcb3f-a771-48a3-bc49-75c332ea3532-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.328867 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.328799 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.329011 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.328902 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.329011 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.328981 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.329084 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.329020 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.429780 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.429740 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.429981 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.429816 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.429981 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.429846 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.429981 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.429875 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.429981 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.429954 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9mr\" (UniqueName: \"kubernetes.io/projected/68ddcb3f-a771-48a3-bc49-75c332ea3532-kube-api-access-rx9mr\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.430201 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.430087 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ddcb3f-a771-48a3-bc49-75c332ea3532-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.430201 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.430127 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.430909 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.430887 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.432285 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.432257 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.432695 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.432638 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ddcb3f-a771-48a3-bc49-75c332ea3532-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.432813 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.432698 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.432813 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.432774 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.452896 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.452867 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ddcb3f-a771-48a3-bc49-75c332ea3532-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.453022 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.452962 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9mr\" (UniqueName: \"kubernetes.io/projected/68ddcb3f-a771-48a3-bc49-75c332ea3532-kube-api-access-rx9mr\") pod \"istiod-openshift-gateway-94bb8fbfd-w9qxn\" (UID: \"68ddcb3f-a771-48a3-bc49-75c332ea3532\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.538474 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.538392 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:31.697214 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.697183 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn"] Jul 09 22:01:31.699951 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:01:31.699923 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ddcb3f_a771_48a3_bc49_75c332ea3532.slice/crio-c0b916e5354479bedbceb257a02b2ce4c29794ba0d97562b21d4399ef294a426 WatchSource:0}: Error finding container c0b916e5354479bedbceb257a02b2ce4c29794ba0d97562b21d4399ef294a426: Status 404 returned error can't find the container with id c0b916e5354479bedbceb257a02b2ce4c29794ba0d97562b21d4399ef294a426 Jul 09 22:01:31.793363 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:31.793279 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" event={"ID":"68ddcb3f-a771-48a3-bc49-75c332ea3532","Type":"ContainerStarted","Data":"c0b916e5354479bedbceb257a02b2ce4c29794ba0d97562b21d4399ef294a426"} Jul 09 22:01:33.096486 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:33.096408 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 22:01:33.096803 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:33.096557 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 22:01:33.096803 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:33.096649 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 22:01:33.807913 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:33.807874 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" event={"ID":"14fc4f3d-f43e-4d5c-839c-902301586b07","Type":"ContainerStarted","Data":"fdb7df33c07d75ca60184b0d33fbf97ff323ac8e0cfbc867fac94f6c290b94ce"} Jul 09 22:01:33.848870 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:33.848818 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" podStartSLOduration=1.534845036 podStartE2EDuration="4.848793897s" totalImagesPullingTime="3.313948861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:01:33.844709147 +0000 UTC m=+613.155326669" watchObservedRunningTime="2026-07-09 22:01:33.848793897 +0000 UTC m=+613.159411423" Jul 09 22:01:34.617964 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:34.617926 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:34.623568 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:34.623538 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:34.818757 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:34.818725 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:34.820101 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:34.820076 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf46nkvx" Jul 09 22:01:34.901704 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:34.901659 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 22:01:34.901848 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:34.901748 2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 09 22:01:35.823778 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:35.823739 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" event={"ID":"68ddcb3f-a771-48a3-bc49-75c332ea3532","Type":"ContainerStarted","Data":"664a9685630de58f02b56f1b7484bb08a9c40949bbdf7093862443fd9d2d8595"} Jul 09 22:01:35.824212 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:35.824134 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:35.825887 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:35.825869 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" Jul 09 22:01:35.864550 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:01:35.864502 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9qxn" podStartSLOduration=1.665403524 podStartE2EDuration="4.864485896s" totalImagesPullingTime="3.199082372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:01:35.862037726 +0000 UTC m=+615.172655253" watchObservedRunningTime="2026-07-09 22:01:35.864485896 +0000 UTC m=+615.175103418" Jul 09 22:02:28.316463 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.316425 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9"] Jul 09 22:02:28.319479 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.319456 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.323431 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.323408 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 22:02:28.324191 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.324176 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-86c5k\"" Jul 09 22:02:28.324248 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.324177 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 22:02:28.329423 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.329399 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9"] Jul 09 22:02:28.348221 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.348191 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gml\" (UniqueName: \"kubernetes.io/projected/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-kube-api-access-p7gml\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.348383 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.348262 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.348383 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.348300 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.449134 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.449102 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.449305 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.449171 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.449305 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.449218 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gml\" (UniqueName: \"kubernetes.io/projected/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-kube-api-access-p7gml\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.449509 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.449491 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.449548 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.449516 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.458233 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.458210 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gml\" (UniqueName: \"kubernetes.io/projected/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-kube-api-access-p7gml\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.629280 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.629250 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:28.756383 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.756357 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9"] Jul 09 22:02:28.758786 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:02:28.758757 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dc060c_16ec_4b9d_9d6b_7ca254ef792c.slice/crio-c31e9d88244ecd00ad30fd29e9d8239e5c8486023898da018c80b3bc06b5dce4 WatchSource:0}: Error finding container c31e9d88244ecd00ad30fd29e9d8239e5c8486023898da018c80b3bc06b5dce4: Status 404 returned error can't find the container with id c31e9d88244ecd00ad30fd29e9d8239e5c8486023898da018c80b3bc06b5dce4 Jul 09 22:02:28.918404 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.918330 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7"] Jul 09 22:02:28.921070 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.921053 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:28.930752 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.930724 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7"] Jul 09 22:02:28.955614 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.955586 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:28.955614 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.955618 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6m9\" (UniqueName: \"kubernetes.io/projected/3271ffea-ab07-40e3-a0b8-5a73e24f920f-kube-api-access-4b6m9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:28.955813 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:28.955701 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:29.057142 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.057109 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:29.057142 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.057142 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6m9\" (UniqueName: \"kubernetes.io/projected/3271ffea-ab07-40e3-a0b8-5a73e24f920f-kube-api-access-4b6m9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:29.057347 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.057185 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:29.057498 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.057479 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:29.057538 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.057506 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:29.065916 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.065892 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6m9\" (UniqueName: \"kubernetes.io/projected/3271ffea-ab07-40e3-a0b8-5a73e24f920f-kube-api-access-4b6m9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:29.080473 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.080446 2556 generic.go:356] "Generic (PLEG): container finished" podID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerID="5d2f0e0a8692067d8de87d3ab311974880c6a4323d6e974a4ceb7e450b6a9c6f" exitCode=0 Jul 09 22:02:29.080603 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.080529 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" event={"ID":"50dc060c-16ec-4b9d-9d6b-7ca254ef792c","Type":"ContainerDied","Data":"5d2f0e0a8692067d8de87d3ab311974880c6a4323d6e974a4ceb7e450b6a9c6f"} Jul 09 22:02:29.080649 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.080605 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" event={"ID":"50dc060c-16ec-4b9d-9d6b-7ca254ef792c","Type":"ContainerStarted","Data":"c31e9d88244ecd00ad30fd29e9d8239e5c8486023898da018c80b3bc06b5dce4"} Jul 09 22:02:29.081053 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.081037 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:02:29.231264 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.231185 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:29.316302 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.316268 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq"] Jul 09 22:02:29.319855 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.319832 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.328347 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.328323 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq"] Jul 09 22:02:29.359976 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.359938 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6jl\" (UniqueName: \"kubernetes.io/projected/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-kube-api-access-hm6jl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.360150 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.360058 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.360150 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.360094 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.377915 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:02:29.377887 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3271ffea_ab07_40e3_a0b8_5a73e24f920f.slice/crio-0883fb0c7a85445833b525a2851898f84beb1d38793f06294c9d043de7d3c43a WatchSource:0}: Error finding container 0883fb0c7a85445833b525a2851898f84beb1d38793f06294c9d043de7d3c43a: Status 404 returned error can't find the container with id 0883fb0c7a85445833b525a2851898f84beb1d38793f06294c9d043de7d3c43a Jul 09 22:02:29.383206 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.383188 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7"] Jul 09 22:02:29.461432 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.461393 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.461432 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.461435 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.461689 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.461540 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6jl\" (UniqueName: \"kubernetes.io/projected/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-kube-api-access-hm6jl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.461904 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.461881 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.461952 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.461894 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.471659 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.471637 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6jl\" (UniqueName: \"kubernetes.io/projected/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-kube-api-access-hm6jl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.634192 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.634166 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:29.719258 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.719226 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv"] Jul 09 22:02:29.722959 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.722937 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.729658 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.729631 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv"] Jul 09 22:02:29.763118 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.763050 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.763256 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.763128 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.763256 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.763167 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2mn\" (UniqueName: \"kubernetes.io/projected/3330a7ab-1f3c-46b8-8ed3-91671393a529-kube-api-access-gr2mn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.770885 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.770861 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq"] Jul 09 22:02:29.773781 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:02:29.773756 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f23b965_be1e_4bfe_bcb2_e7c68237fee8.slice/crio-f9596db7d5f095fc5e94732da50692d2440f8e1c1854a87e7536bd167c2347be WatchSource:0}: Error finding container f9596db7d5f095fc5e94732da50692d2440f8e1c1854a87e7536bd167c2347be: Status 404 returned error can't find the container with id f9596db7d5f095fc5e94732da50692d2440f8e1c1854a87e7536bd167c2347be Jul 09 22:02:29.864791 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.864756 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.864958 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.864818 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2mn\" (UniqueName: \"kubernetes.io/projected/3330a7ab-1f3c-46b8-8ed3-91671393a529-kube-api-access-gr2mn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.864958 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.864913 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.865189 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.865168 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.865226 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.865191 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:29.873833 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:29.873807 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2mn\" (UniqueName: \"kubernetes.io/projected/3330a7ab-1f3c-46b8-8ed3-91671393a529-kube-api-access-gr2mn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:30.035519 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.035491 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:30.088321 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.088273 2556 generic.go:356] "Generic (PLEG): container finished" podID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerID="5ce3351d49b1299234ed58600948b98764504379ece20428bbe26938811f2e30" exitCode=0 Jul 09 22:02:30.088458 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.088429 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" event={"ID":"3271ffea-ab07-40e3-a0b8-5a73e24f920f","Type":"ContainerDied","Data":"5ce3351d49b1299234ed58600948b98764504379ece20428bbe26938811f2e30"} Jul 09 22:02:30.088528 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.088480 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" event={"ID":"3271ffea-ab07-40e3-a0b8-5a73e24f920f","Type":"ContainerStarted","Data":"0883fb0c7a85445833b525a2851898f84beb1d38793f06294c9d043de7d3c43a"} Jul 09 22:02:30.091483 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.091411 2556 generic.go:356] "Generic (PLEG): container finished" podID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerID="2ef847ffc15245bb1bf8ae6e0f7a5b7da6dd5d78870ef16bc994d9f20f281330" exitCode=0 Jul 09 22:02:30.091612 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.091510 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" event={"ID":"50dc060c-16ec-4b9d-9d6b-7ca254ef792c","Type":"ContainerDied","Data":"2ef847ffc15245bb1bf8ae6e0f7a5b7da6dd5d78870ef16bc994d9f20f281330"} Jul 09 22:02:30.095531 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.095384 2556 generic.go:356] "Generic (PLEG): container finished" podID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerID="74239cfdf7ae3e17cdba438936da62fa0481fc9f606f8673c2728fcb79b17e50" exitCode=0 Jul 09 22:02:30.095531 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.095464 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" event={"ID":"8f23b965-be1e-4bfe-bcb2-e7c68237fee8","Type":"ContainerDied","Data":"74239cfdf7ae3e17cdba438936da62fa0481fc9f606f8673c2728fcb79b17e50"} Jul 09 22:02:30.095531 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.095503 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" event={"ID":"8f23b965-be1e-4bfe-bcb2-e7c68237fee8","Type":"ContainerStarted","Data":"f9596db7d5f095fc5e94732da50692d2440f8e1c1854a87e7536bd167c2347be"} Jul 09 22:02:30.179149 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:30.179126 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv"] Jul 09 22:02:30.180802 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:02:30.180772 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3330a7ab_1f3c_46b8_8ed3_91671393a529.slice/crio-3f28e5bfca31cdd2121d03d18fa875381a55406f3077e5f313bf5ec6f7cbdd53 WatchSource:0}: Error finding container 3f28e5bfca31cdd2121d03d18fa875381a55406f3077e5f313bf5ec6f7cbdd53: Status 404 returned error can't find the container with id 3f28e5bfca31cdd2121d03d18fa875381a55406f3077e5f313bf5ec6f7cbdd53 Jul 09 22:02:31.102420 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.102393 2556 generic.go:356] "Generic (PLEG): container finished" podID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerID="290c16b90ba40b3e5763c28b0b346e02a10561fcadf5baf9255b9e50fd40cf9f" exitCode=0 Jul 09 22:02:31.102861 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.102469 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" event={"ID":"3271ffea-ab07-40e3-a0b8-5a73e24f920f","Type":"ContainerDied","Data":"290c16b90ba40b3e5763c28b0b346e02a10561fcadf5baf9255b9e50fd40cf9f"} Jul 09 22:02:31.104035 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.104006 2556 generic.go:356] "Generic (PLEG): container finished" podID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerID="c06cc44913b66dba45b14ebac54e44823815815ba144ace9847b69f677e0d568" exitCode=0 Jul 09 22:02:31.104160 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.104039 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" event={"ID":"3330a7ab-1f3c-46b8-8ed3-91671393a529","Type":"ContainerDied","Data":"c06cc44913b66dba45b14ebac54e44823815815ba144ace9847b69f677e0d568"} Jul 09 22:02:31.104160 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.104088 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" event={"ID":"3330a7ab-1f3c-46b8-8ed3-91671393a529","Type":"ContainerStarted","Data":"3f28e5bfca31cdd2121d03d18fa875381a55406f3077e5f313bf5ec6f7cbdd53"} Jul 09 22:02:31.106291 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.106267 2556 generic.go:356] "Generic (PLEG): container finished" podID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerID="9ec5a40af1daa1735f58f38f44f57e7642aba0872d10cd83e45287b10d9538aa" exitCode=0 Jul 09 22:02:31.106400 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.106357 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" event={"ID":"50dc060c-16ec-4b9d-9d6b-7ca254ef792c","Type":"ContainerDied","Data":"9ec5a40af1daa1735f58f38f44f57e7642aba0872d10cd83e45287b10d9538aa"} Jul 09 22:02:31.108522 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.108505 2556 generic.go:356] "Generic (PLEG): container finished" podID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerID="9366ddac505cee6fa142fc76bff18162b90f657a80fa0ece09f1bebb610ab9f7" exitCode=0 Jul 09 22:02:31.108709 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:31.108593 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" event={"ID":"8f23b965-be1e-4bfe-bcb2-e7c68237fee8","Type":"ContainerDied","Data":"9366ddac505cee6fa142fc76bff18162b90f657a80fa0ece09f1bebb610ab9f7"} Jul 09 22:02:32.116507 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.116480 2556 generic.go:356] "Generic (PLEG): container finished" podID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerID="23bc43436715eb7ee207209799d6c709e911e2bd5182d00431ffcb4642aa6bab" exitCode=0 Jul 09 22:02:32.116915 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.116557 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" event={"ID":"3271ffea-ab07-40e3-a0b8-5a73e24f920f","Type":"ContainerDied","Data":"23bc43436715eb7ee207209799d6c709e911e2bd5182d00431ffcb4642aa6bab"} Jul 09 22:02:32.118301 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.118282 2556 generic.go:356] "Generic (PLEG): container finished" podID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerID="14b6f7546a5488ca1792c6e1b2f3b200c6e01e27e2054117a3a18a8dcd760699" exitCode=0 Jul 09 22:02:32.118415 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.118368 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" event={"ID":"3330a7ab-1f3c-46b8-8ed3-91671393a529","Type":"ContainerDied","Data":"14b6f7546a5488ca1792c6e1b2f3b200c6e01e27e2054117a3a18a8dcd760699"} Jul 09 22:02:32.120492 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.120472 2556 generic.go:356] "Generic (PLEG): container finished" podID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerID="9a607cc67d56eb2bf2e122b6e91af8bef9b5faede464f448aff18b5183b949a1" exitCode=0 Jul 09 22:02:32.120619 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.120535 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" event={"ID":"8f23b965-be1e-4bfe-bcb2-e7c68237fee8","Type":"ContainerDied","Data":"9a607cc67d56eb2bf2e122b6e91af8bef9b5faede464f448aff18b5183b949a1"} Jul 09 22:02:32.253325 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.253305 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:32.287606 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.287569 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-bundle\") pod \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " Jul 09 22:02:32.287732 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.287632 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7gml\" (UniqueName: \"kubernetes.io/projected/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-kube-api-access-p7gml\") pod \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " Jul 09 22:02:32.287732 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.287659 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-util\") pod \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\" (UID: \"50dc060c-16ec-4b9d-9d6b-7ca254ef792c\") " Jul 09 22:02:32.288106 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.288083 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-bundle" (OuterVolumeSpecName: "bundle") pod "50dc060c-16ec-4b9d-9d6b-7ca254ef792c" (UID: "50dc060c-16ec-4b9d-9d6b-7ca254ef792c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:02:32.289847 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.289824 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-kube-api-access-p7gml" (OuterVolumeSpecName: "kube-api-access-p7gml") pod "50dc060c-16ec-4b9d-9d6b-7ca254ef792c" (UID: "50dc060c-16ec-4b9d-9d6b-7ca254ef792c"). InnerVolumeSpecName "kube-api-access-p7gml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:02:32.293061 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.293039 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-util" (OuterVolumeSpecName: "util") pod "50dc060c-16ec-4b9d-9d6b-7ca254ef792c" (UID: "50dc060c-16ec-4b9d-9d6b-7ca254ef792c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:02:32.388640 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.388605 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-bundle\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:32.388640 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.388636 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7gml\" (UniqueName: \"kubernetes.io/projected/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-kube-api-access-p7gml\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:32.388640 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:32.388646 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50dc060c-16ec-4b9d-9d6b-7ca254ef792c-util\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:33.127422 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.127392 2556 generic.go:356] "Generic (PLEG): container finished" podID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerID="35d03653062077e20c15cc75f8d1ffbf4b36e9423db4997c517d886c86f9d76c" exitCode=0 Jul 09 22:02:33.127828 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.127479 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" event={"ID":"3330a7ab-1f3c-46b8-8ed3-91671393a529","Type":"ContainerDied","Data":"35d03653062077e20c15cc75f8d1ffbf4b36e9423db4997c517d886c86f9d76c"} Jul 09 22:02:33.129318 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.129301 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" Jul 09 22:02:33.129403 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.129326 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9" event={"ID":"50dc060c-16ec-4b9d-9d6b-7ca254ef792c","Type":"ContainerDied","Data":"c31e9d88244ecd00ad30fd29e9d8239e5c8486023898da018c80b3bc06b5dce4"} Jul 09 22:02:33.129403 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.129358 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31e9d88244ecd00ad30fd29e9d8239e5c8486023898da018c80b3bc06b5dce4" Jul 09 22:02:33.290925 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.290902 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:33.293402 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.293380 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:33.395539 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.395447 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-bundle\") pod \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " Jul 09 22:02:33.395539 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.395485 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm6jl\" (UniqueName: \"kubernetes.io/projected/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-kube-api-access-hm6jl\") pod \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " Jul 09 22:02:33.395781 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.395600 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-util\") pod \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " Jul 09 22:02:33.395781 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.395642 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b6m9\" (UniqueName: \"kubernetes.io/projected/3271ffea-ab07-40e3-a0b8-5a73e24f920f-kube-api-access-4b6m9\") pod \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\" (UID: \"3271ffea-ab07-40e3-a0b8-5a73e24f920f\") " Jul 09 22:02:33.395781 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.395695 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-util\") pod \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " Jul 09 22:02:33.395781 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.395723 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-bundle\") pod \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\" (UID: \"8f23b965-be1e-4bfe-bcb2-e7c68237fee8\") " Jul 09 22:02:33.396216 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.396182 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-bundle" (OuterVolumeSpecName: "bundle") pod "3271ffea-ab07-40e3-a0b8-5a73e24f920f" (UID: "3271ffea-ab07-40e3-a0b8-5a73e24f920f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:02:33.396604 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.396560 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-bundle" (OuterVolumeSpecName: "bundle") pod "8f23b965-be1e-4bfe-bcb2-e7c68237fee8" (UID: "8f23b965-be1e-4bfe-bcb2-e7c68237fee8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:02:33.397859 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.397839 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-kube-api-access-hm6jl" (OuterVolumeSpecName: "kube-api-access-hm6jl") pod "8f23b965-be1e-4bfe-bcb2-e7c68237fee8" (UID: "8f23b965-be1e-4bfe-bcb2-e7c68237fee8"). InnerVolumeSpecName "kube-api-access-hm6jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:02:33.397970 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.397951 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3271ffea-ab07-40e3-a0b8-5a73e24f920f-kube-api-access-4b6m9" (OuterVolumeSpecName: "kube-api-access-4b6m9") pod "3271ffea-ab07-40e3-a0b8-5a73e24f920f" (UID: "3271ffea-ab07-40e3-a0b8-5a73e24f920f"). InnerVolumeSpecName "kube-api-access-4b6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:02:33.404346 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.404317 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-util" (OuterVolumeSpecName: "util") pod "3271ffea-ab07-40e3-a0b8-5a73e24f920f" (UID: "3271ffea-ab07-40e3-a0b8-5a73e24f920f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:02:33.404642 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.404569 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-util" (OuterVolumeSpecName: "util") pod "8f23b965-be1e-4bfe-bcb2-e7c68237fee8" (UID: "8f23b965-be1e-4bfe-bcb2-e7c68237fee8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:02:33.496905 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.496869 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-bundle\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:33.496905 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.496897 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6jl\" (UniqueName: \"kubernetes.io/projected/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-kube-api-access-hm6jl\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:33.496905 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.496909 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3271ffea-ab07-40e3-a0b8-5a73e24f920f-util\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:33.497143 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.496918 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b6m9\" (UniqueName: \"kubernetes.io/projected/3271ffea-ab07-40e3-a0b8-5a73e24f920f-kube-api-access-4b6m9\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:33.497143 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.496927 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-util\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:33.497143 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:33.496934 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f23b965-be1e-4bfe-bcb2-e7c68237fee8-bundle\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:34.137367 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.137337 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" Jul 09 22:02:34.137367 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.137356 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq" event={"ID":"8f23b965-be1e-4bfe-bcb2-e7c68237fee8","Type":"ContainerDied","Data":"f9596db7d5f095fc5e94732da50692d2440f8e1c1854a87e7536bd167c2347be"} Jul 09 22:02:34.137869 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.137404 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9596db7d5f095fc5e94732da50692d2440f8e1c1854a87e7536bd167c2347be" Jul 09 22:02:34.139299 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.139276 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" Jul 09 22:02:34.139419 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.139315 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7" event={"ID":"3271ffea-ab07-40e3-a0b8-5a73e24f920f","Type":"ContainerDied","Data":"0883fb0c7a85445833b525a2851898f84beb1d38793f06294c9d043de7d3c43a"} Jul 09 22:02:34.139419 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.139348 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0883fb0c7a85445833b525a2851898f84beb1d38793f06294c9d043de7d3c43a" Jul 09 22:02:34.265819 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.265790 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:02:34.303042 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.303015 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr2mn\" (UniqueName: \"kubernetes.io/projected/3330a7ab-1f3c-46b8-8ed3-91671393a529-kube-api-access-gr2mn\") pod \"3330a7ab-1f3c-46b8-8ed3-91671393a529\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " Jul 09 22:02:34.303198 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.303068 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-util\") pod \"3330a7ab-1f3c-46b8-8ed3-91671393a529\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " Jul 09 22:02:34.303252 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.303191 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-bundle\") pod \"3330a7ab-1f3c-46b8-8ed3-91671393a529\" (UID: \"3330a7ab-1f3c-46b8-8ed3-91671393a529\") " Jul 09 22:02:34.303608 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.303565 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-bundle" (OuterVolumeSpecName: "bundle") pod "3330a7ab-1f3c-46b8-8ed3-91671393a529" (UID: "3330a7ab-1f3c-46b8-8ed3-91671393a529"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:02:34.305614 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.305595 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3330a7ab-1f3c-46b8-8ed3-91671393a529-kube-api-access-gr2mn" (OuterVolumeSpecName: "kube-api-access-gr2mn") pod "3330a7ab-1f3c-46b8-8ed3-91671393a529" (UID: "3330a7ab-1f3c-46b8-8ed3-91671393a529"). InnerVolumeSpecName "kube-api-access-gr2mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:02:34.309069 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.309046 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-util" (OuterVolumeSpecName: "util") pod "3330a7ab-1f3c-46b8-8ed3-91671393a529" (UID: "3330a7ab-1f3c-46b8-8ed3-91671393a529"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:02:34.404166 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.404085 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr2mn\" (UniqueName: \"kubernetes.io/projected/3330a7ab-1f3c-46b8-8ed3-91671393a529-kube-api-access-gr2mn\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:34.404166 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.404109 2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-util\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:34.404166 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:34.404120 2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3330a7ab-1f3c-46b8-8ed3-91671393a529-bundle\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:02:35.147142 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:35.147113 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" event={"ID":"3330a7ab-1f3c-46b8-8ed3-91671393a529","Type":"ContainerDied","Data":"3f28e5bfca31cdd2121d03d18fa875381a55406f3077e5f313bf5ec6f7cbdd53"} Jul 09 22:02:35.147507 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:35.147158 2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f28e5bfca31cdd2121d03d18fa875381a55406f3077e5f313bf5ec6f7cbdd53" Jul 09 22:02:35.147507 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:02:35.147159 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv" Jul 09 22:04:41.796863 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.796826 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85949c664f-d77n9"] Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797227 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerName="pull" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797241 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerName="pull" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797257 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerName="util" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797263 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerName="util" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797270 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerName="util" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797275 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerName="util" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797281 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerName="pull" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797286 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerName="pull" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797292 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerName="extract" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797297 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerName="extract" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797306 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerName="util" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797314 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerName="util" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797322 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerName="extract" Jul 09 22:04:41.797333 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797326 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerName="extract" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797347 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerName="extract" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797353 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerName="extract" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797358 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerName="pull" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797363 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerName="pull" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797368 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerName="extract" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797373 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerName="extract" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797386 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerName="util" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797395 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerName="util" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797403 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerName="pull" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797408 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerName="pull" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797474 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f23b965-be1e-4bfe-bcb2-e7c68237fee8" containerName="extract" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797483 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3330a7ab-1f3c-46b8-8ed3-91671393a529" containerName="extract" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797488 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="50dc060c-16ec-4b9d-9d6b-7ca254ef792c" containerName="extract" Jul 09 22:04:41.797775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.797496 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3271ffea-ab07-40e3-a0b8-5a73e24f920f" containerName="extract" Jul 09 22:04:41.800035 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.800021 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:41.802659 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.802630 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 22:04:41.802789 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.802640 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 22:04:41.802789 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.802673 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddbdk\"" Jul 09 22:04:41.810418 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.810395 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85949c664f-d77n9"] Jul 09 22:04:41.938255 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.938219 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert\") pod \"maas-controller-85949c664f-d77n9\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:41.938438 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.938276 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9md2\" (UniqueName: \"kubernetes.io/projected/d4fd8a5c-a869-439d-a039-9467c67e8cfc-kube-api-access-g9md2\") pod \"maas-controller-85949c664f-d77n9\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:41.982757 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.982725 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-768b466dcc-jr4rl"] Jul 09 22:04:41.985393 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.985372 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:41.995531 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:41.995503 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768b466dcc-jr4rl"] Jul 09 22:04:42.039591 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.039538 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9md2\" (UniqueName: \"kubernetes.io/projected/d4fd8a5c-a869-439d-a039-9467c67e8cfc-kube-api-access-g9md2\") pod \"maas-controller-85949c664f-d77n9\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:42.039751 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.039650 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert\") pod \"maas-controller-85949c664f-d77n9\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:42.039751 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:04:42.039747 2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 09 22:04:42.039823 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:04:42.039791 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert podName:d4fd8a5c-a869-439d-a039-9467c67e8cfc nodeName:}" failed. No retries permitted until 2026-07-09 22:04:42.539776724 +0000 UTC m=+801.850394226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert") pod "maas-controller-85949c664f-d77n9" (UID: "d4fd8a5c-a869-439d-a039-9467c67e8cfc") : secret "maas-controller-webhook-cert" not found Jul 09 22:04:42.051349 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.051287 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9md2\" (UniqueName: \"kubernetes.io/projected/d4fd8a5c-a869-439d-a039-9467c67e8cfc-kube-api-access-g9md2\") pod \"maas-controller-85949c664f-d77n9\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:42.140366 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.140328 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-webhook-cert\") pod \"maas-controller-768b466dcc-jr4rl\" (UID: \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\") " pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:42.140540 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.140392 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzrw\" (UniqueName: \"kubernetes.io/projected/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-kube-api-access-hbzrw\") pod \"maas-controller-768b466dcc-jr4rl\" (UID: \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\") " pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:42.241051 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.240973 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-webhook-cert\") pod \"maas-controller-768b466dcc-jr4rl\" (UID: \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\") " pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:42.241051 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.241032 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzrw\" (UniqueName: \"kubernetes.io/projected/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-kube-api-access-hbzrw\") pod \"maas-controller-768b466dcc-jr4rl\" (UID: \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\") " pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:42.243341 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.243312 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-webhook-cert\") pod \"maas-controller-768b466dcc-jr4rl\" (UID: \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\") " pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:42.249929 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.249905 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzrw\" (UniqueName: \"kubernetes.io/projected/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-kube-api-access-hbzrw\") pod \"maas-controller-768b466dcc-jr4rl\" (UID: \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\") " pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:42.296482 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.296450 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:42.544007 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.543968 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert\") pod \"maas-controller-85949c664f-d77n9\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:42.546420 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.546390 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert\") pod \"maas-controller-85949c664f-d77n9\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:42.635986 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.635962 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768b466dcc-jr4rl"] Jul 09 22:04:42.638045 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:04:42.638021 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ca2676_c34a_4554_bc4a_0ed5ddf13e8c.slice/crio-3ab943d43e32000c0612dba7655e39ff4ea029fdde19db77fb324a967c48ceaa WatchSource:0}: Error finding container 3ab943d43e32000c0612dba7655e39ff4ea029fdde19db77fb324a967c48ceaa: Status 404 returned error can't find the container with id 3ab943d43e32000c0612dba7655e39ff4ea029fdde19db77fb324a967c48ceaa Jul 09 22:04:42.711214 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.711176 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:42.805471 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.805428 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768b466dcc-jr4rl" event={"ID":"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c","Type":"ContainerStarted","Data":"3ab943d43e32000c0612dba7655e39ff4ea029fdde19db77fb324a967c48ceaa"} Jul 09 22:04:42.844370 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:42.844349 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85949c664f-d77n9"] Jul 09 22:04:42.846185 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:04:42.846157 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fd8a5c_a869_439d_a039_9467c67e8cfc.slice/crio-78ab272c4eb5ae91d0dabf62ff6c3e286cb24e3e8ff8ab931ae36fc57d90cc82 WatchSource:0}: Error finding container 78ab272c4eb5ae91d0dabf62ff6c3e286cb24e3e8ff8ab931ae36fc57d90cc82: Status 404 returned error can't find the container with id 78ab272c4eb5ae91d0dabf62ff6c3e286cb24e3e8ff8ab931ae36fc57d90cc82 Jul 09 22:04:43.818894 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:43.818856 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85949c664f-d77n9" event={"ID":"d4fd8a5c-a869-439d-a039-9467c67e8cfc","Type":"ContainerStarted","Data":"78ab272c4eb5ae91d0dabf62ff6c3e286cb24e3e8ff8ab931ae36fc57d90cc82"} Jul 09 22:04:45.830603 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:45.830548 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768b466dcc-jr4rl" event={"ID":"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c","Type":"ContainerStarted","Data":"eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6"} Jul 09 22:04:45.831040 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:45.830664 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:45.832238 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:45.832211 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85949c664f-d77n9" event={"ID":"d4fd8a5c-a869-439d-a039-9467c67e8cfc","Type":"ContainerStarted","Data":"0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7"} Jul 09 22:04:45.832366 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:45.832352 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:45.889890 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:45.889842 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-768b466dcc-jr4rl" podStartSLOduration=2.40597933 podStartE2EDuration="4.889828087s" totalImagesPullingTime="2.483848757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:04:45.886910553 +0000 UTC m=+805.197528077" watchObservedRunningTime="2026-07-09 22:04:45.889828087 +0000 UTC m=+805.200445611" Jul 09 22:04:45.923757 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:45.923708 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-85949c664f-d77n9" podStartSLOduration=2.659079764 podStartE2EDuration="4.923690026s" totalImagesPullingTime="2.264610262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:04:45.923148992 +0000 UTC m=+805.233766515" watchObservedRunningTime="2026-07-09 22:04:45.923690026 +0000 UTC m=+805.234307559" Jul 09 22:04:56.843944 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:56.843913 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:56.844544 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:56.843988 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:04:56.907656 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:56.907626 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85949c664f-d77n9"] Jul 09 22:04:56.907877 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:56.907837 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-85949c664f-d77n9" podUID="d4fd8a5c-a869-439d-a039-9467c67e8cfc" containerName="manager" containerID="cri-o://0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7" gracePeriod=30 Jul 09 22:04:57.155511 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.155482 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:57.280244 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.280216 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert\") pod \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " Jul 09 22:04:57.280455 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.280372 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9md2\" (UniqueName: \"kubernetes.io/projected/d4fd8a5c-a869-439d-a039-9467c67e8cfc-kube-api-access-g9md2\") pod \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\" (UID: \"d4fd8a5c-a869-439d-a039-9467c67e8cfc\") " Jul 09 22:04:57.282635 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.282566 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fd8a5c-a869-439d-a039-9467c67e8cfc-kube-api-access-g9md2" (OuterVolumeSpecName: "kube-api-access-g9md2") pod "d4fd8a5c-a869-439d-a039-9467c67e8cfc" (UID: "d4fd8a5c-a869-439d-a039-9467c67e8cfc"). InnerVolumeSpecName "kube-api-access-g9md2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:04:57.282635 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.282596 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d4fd8a5c-a869-439d-a039-9467c67e8cfc" (UID: "d4fd8a5c-a869-439d-a039-9467c67e8cfc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:04:57.381204 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.381171 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9md2\" (UniqueName: \"kubernetes.io/projected/d4fd8a5c-a869-439d-a039-9467c67e8cfc-kube-api-access-g9md2\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:04:57.381204 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.381200 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4fd8a5c-a869-439d-a039-9467c67e8cfc-webhook-cert\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:04:57.894851 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.894820 2556 generic.go:356] "Generic (PLEG): container finished" podID="d4fd8a5c-a869-439d-a039-9467c67e8cfc" containerID="0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7" exitCode=0 Jul 09 22:04:57.895242 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.894873 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85949c664f-d77n9" Jul 09 22:04:57.895242 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.894877 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85949c664f-d77n9" event={"ID":"d4fd8a5c-a869-439d-a039-9467c67e8cfc","Type":"ContainerDied","Data":"0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7"} Jul 09 22:04:57.895242 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.894933 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85949c664f-d77n9" event={"ID":"d4fd8a5c-a869-439d-a039-9467c67e8cfc","Type":"ContainerDied","Data":"78ab272c4eb5ae91d0dabf62ff6c3e286cb24e3e8ff8ab931ae36fc57d90cc82"} Jul 09 22:04:57.895242 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.894958 2556 scope.go:117] "RemoveContainer" containerID="0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7" Jul 09 22:04:57.904470 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.904455 2556 scope.go:117] "RemoveContainer" containerID="0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7" Jul 09 22:04:57.904742 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:04:57.904723 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7\": container with ID starting with 0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7 not found: ID does not exist" containerID="0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7" Jul 09 22:04:57.904827 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.904756 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7"} err="failed to get container status \"0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7\": rpc error: code = NotFound desc = could not find container \"0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7\": container with ID starting with 0fb5ab4a7d87bb01ab2333a1ffb824d8e2331c9dd3fcaa5c2856c7641daaa6d7 not found: ID does not exist" Jul 09 22:04:57.915890 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.915865 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85949c664f-d77n9"] Jul 09 22:04:57.919940 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:57.919916 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85949c664f-d77n9"] Jul 09 22:04:59.275690 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:04:59.275657 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4fd8a5c-a869-439d-a039-9467c67e8cfc" path="/var/lib/kubelet/pods/d4fd8a5c-a869-439d-a039-9467c67e8cfc/volumes" Jul 09 22:05:17.417341 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.417301 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749"] Jul 09 22:05:17.417962 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.417896 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fd8a5c-a869-439d-a039-9467c67e8cfc" containerName="manager" Jul 09 22:05:17.417962 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.417915 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fd8a5c-a869-439d-a039-9467c67e8cfc" containerName="manager" Jul 09 22:05:17.418079 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.418017 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fd8a5c-a869-439d-a039-9467c67e8cfc" containerName="manager" Jul 09 22:05:17.420910 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.420891 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.423406 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.423386 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 22:05:17.423507 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.423387 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:05:17.423507 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.423387 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 22:05:17.423507 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.423421 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-t4ml9\"" Jul 09 22:05:17.432297 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.432279 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749"] Jul 09 22:05:17.444846 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.444820 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1469f213-0b26-43aa-9ac2-ea55633821f0-openshift-service-ca\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.444947 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.444871 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1469f213-0b26-43aa-9ac2-ea55633821f0-maas-api-tls\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.444947 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.444896 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-924bp\" (UniqueName: \"kubernetes.io/projected/1469f213-0b26-43aa-9ac2-ea55633821f0-kube-api-access-924bp\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.546094 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.546058 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1469f213-0b26-43aa-9ac2-ea55633821f0-openshift-service-ca\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.546094 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.546101 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1469f213-0b26-43aa-9ac2-ea55633821f0-maas-api-tls\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.546356 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.546120 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-924bp\" (UniqueName: \"kubernetes.io/projected/1469f213-0b26-43aa-9ac2-ea55633821f0-kube-api-access-924bp\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.546777 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.546753 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1469f213-0b26-43aa-9ac2-ea55633821f0-openshift-service-ca\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.548419 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.548399 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1469f213-0b26-43aa-9ac2-ea55633821f0-maas-api-tls\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.561933 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.561912 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-924bp\" (UniqueName: \"kubernetes.io/projected/1469f213-0b26-43aa-9ac2-ea55633821f0-kube-api-access-924bp\") pod \"maas-api-6b7f5f6db5-5x749\" (UID: \"1469f213-0b26-43aa-9ac2-ea55633821f0\") " pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.731727 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.731648 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:17.873807 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:17.873776 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749"] Jul 09 22:05:17.875469 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:05:17.875446 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1469f213_0b26_43aa_9ac2_ea55633821f0.slice/crio-6d0f4fdb397cd074c1945a379f0ce2a3927eb5dfdeea1a772cff4ec226eb7252 WatchSource:0}: Error finding container 6d0f4fdb397cd074c1945a379f0ce2a3927eb5dfdeea1a772cff4ec226eb7252: Status 404 returned error can't find the container with id 6d0f4fdb397cd074c1945a379f0ce2a3927eb5dfdeea1a772cff4ec226eb7252 Jul 09 22:05:18.003923 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.003843 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" event={"ID":"1469f213-0b26-43aa-9ac2-ea55633821f0","Type":"ContainerStarted","Data":"6d0f4fdb397cd074c1945a379f0ce2a3927eb5dfdeea1a772cff4ec226eb7252"} Jul 09 22:05:18.026601 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.026548 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-zg8bh"] Jul 09 22:05:18.031027 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.031010 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:18.033457 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.033436 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gd7z9\"" Jul 09 22:05:18.033558 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.033438 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 22:05:18.037720 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.037698 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-zg8bh"] Jul 09 22:05:18.151840 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.151807 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6p9m\" (UniqueName: \"kubernetes.io/projected/89d5f6cf-36e8-4565-ae62-d2e7e28c9af0-kube-api-access-p6p9m\") pod \"payload-processing-58f86c7798-zg8bh\" (UID: \"89d5f6cf-36e8-4565-ae62-d2e7e28c9af0\") " pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:18.152011 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.151846 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d5f6cf-36e8-4565-ae62-d2e7e28c9af0-plugins-config-volume\") pod \"payload-processing-58f86c7798-zg8bh\" (UID: \"89d5f6cf-36e8-4565-ae62-d2e7e28c9af0\") " pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:18.253131 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.253091 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6p9m\" (UniqueName: \"kubernetes.io/projected/89d5f6cf-36e8-4565-ae62-d2e7e28c9af0-kube-api-access-p6p9m\") pod \"payload-processing-58f86c7798-zg8bh\" (UID: \"89d5f6cf-36e8-4565-ae62-d2e7e28c9af0\") " pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:18.253314 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.253136 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d5f6cf-36e8-4565-ae62-d2e7e28c9af0-plugins-config-volume\") pod \"payload-processing-58f86c7798-zg8bh\" (UID: \"89d5f6cf-36e8-4565-ae62-d2e7e28c9af0\") " pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:18.253906 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.253882 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89d5f6cf-36e8-4565-ae62-d2e7e28c9af0-plugins-config-volume\") pod \"payload-processing-58f86c7798-zg8bh\" (UID: \"89d5f6cf-36e8-4565-ae62-d2e7e28c9af0\") " pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:18.261694 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.261641 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6p9m\" (UniqueName: \"kubernetes.io/projected/89d5f6cf-36e8-4565-ae62-d2e7e28c9af0-kube-api-access-p6p9m\") pod \"payload-processing-58f86c7798-zg8bh\" (UID: \"89d5f6cf-36e8-4565-ae62-d2e7e28c9af0\") " pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:18.343599 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.343538 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:18.481965 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:18.481936 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-zg8bh"] Jul 09 22:05:18.483090 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:05:18.483066 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d5f6cf_36e8_4565_ae62_d2e7e28c9af0.slice/crio-865073f550fa5835efe9a8caf441182c5e0740ae26acfbf67c13056ca2ec653c WatchSource:0}: Error finding container 865073f550fa5835efe9a8caf441182c5e0740ae26acfbf67c13056ca2ec653c: Status 404 returned error can't find the container with id 865073f550fa5835efe9a8caf441182c5e0740ae26acfbf67c13056ca2ec653c Jul 09 22:05:19.009745 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:19.009711 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" event={"ID":"89d5f6cf-36e8-4565-ae62-d2e7e28c9af0","Type":"ContainerStarted","Data":"865073f550fa5835efe9a8caf441182c5e0740ae26acfbf67c13056ca2ec653c"} Jul 09 22:05:20.017142 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:20.017103 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" event={"ID":"89d5f6cf-36e8-4565-ae62-d2e7e28c9af0","Type":"ContainerStarted","Data":"441da65090522af305d64b2f727dc41c440eabf02b976621dad6255414e6f3df"} Jul 09 22:05:20.017653 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:20.017203 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:05:20.035330 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:20.035243 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" podStartSLOduration=0.591251074 podStartE2EDuration="2.035223092s" totalImagesPullingTime="1.443972018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:05:20.034454259 +0000 UTC m=+839.345071783" watchObservedRunningTime="2026-07-09 22:05:20.035223092 +0000 UTC m=+839.345840596" Jul 09 22:05:22.028973 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:22.028942 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" event={"ID":"1469f213-0b26-43aa-9ac2-ea55633821f0","Type":"ContainerStarted","Data":"ea1579d0e4af0670b1bb5d091a1ae24bf7ea1f70d4fe7f0467b86d6d440d3415"} Jul 09 22:05:22.029404 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:22.029076 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:22.047875 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:22.047826 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" podStartSLOduration=1.425474721 podStartE2EDuration="5.047811898s" totalImagesPullingTime="3.622337177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:05:22.044949127 +0000 UTC m=+841.355566650" watchObservedRunningTime="2026-07-09 22:05:22.047811898 +0000 UTC m=+841.358429420" Jul 09 22:05:28.040908 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:28.040877 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b7f5f6db5-5x749" Jul 09 22:05:31.025206 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:05:31.025169 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-zg8bh" Jul 09 22:06:09.591938 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.591890 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz"] Jul 09 22:06:09.595420 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.595382 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.598998 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.598976 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 22:06:09.599134 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.599079 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x55j4\"" Jul 09 22:06:09.599134 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.599079 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 09 22:06:09.599134 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.599081 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 22:06:09.607832 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.607808 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz"] Jul 09 22:06:09.717768 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.717713 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.717952 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.717876 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.718015 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.717974 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.718067 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.718018 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qb4x\" (UniqueName: \"kubernetes.io/projected/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-kube-api-access-6qb4x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.718067 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.718054 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.718172 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.718114 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.819363 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.819329 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.819363 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.819367 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.819652 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.819416 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.819652 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.819445 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.819652 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.819469 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qb4x\" (UniqueName: \"kubernetes.io/projected/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-kube-api-access-6qb4x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.819652 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.819491 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.819882 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.819842 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.819882 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.819854 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.820025 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.820004 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.821780 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.821760 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.822106 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.822090 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.827036 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.827013 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qb4x\" (UniqueName: \"kubernetes.io/projected/6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c-kube-api-access-6qb4x\") pod \"e2e-distinct-simulated-kserve-59975d69f9-fhcqz\" (UID: \"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:09.927434 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:09.927352 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:10.060454 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:10.060427 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz"] Jul 09 22:06:10.061927 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:06:10.061900 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5ef9ab_78d0_4fa3_afbc_c107879f6c1c.slice/crio-10fac85e7cda6eee5c1d95764a4c768d723f53deb3242132937efbb59c3cec7e WatchSource:0}: Error finding container 10fac85e7cda6eee5c1d95764a4c768d723f53deb3242132937efbb59c3cec7e: Status 404 returned error can't find the container with id 10fac85e7cda6eee5c1d95764a4c768d723f53deb3242132937efbb59c3cec7e Jul 09 22:06:10.293416 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:10.293339 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" event={"ID":"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c","Type":"ContainerStarted","Data":"10fac85e7cda6eee5c1d95764a4c768d723f53deb3242132937efbb59c3cec7e"} Jul 09 22:06:11.300182 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:11.300148 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" event={"ID":"6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c","Type":"ContainerStarted","Data":"a70beebeb7bca61a95ab7aff15ac1f2182f33d49838d34154f9da1bbff368206"} Jul 09 22:06:11.319715 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:11.319665 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" podStartSLOduration=1.29876269 podStartE2EDuration="2.319650458s" totalImagesPullingTime="1.020887768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:06:11.318649872 +0000 UTC m=+890.629267396" watchObservedRunningTime="2026-07-09 22:06:11.319650458 +0000 UTC m=+890.630267981" Jul 09 22:06:19.927878 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:19.927784 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:19.928348 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:19.927893 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:19.940684 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:19.940661 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:20.360769 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:20.360744 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-fhcqz" Jul 09 22:06:21.175356 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:21.175328 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:06:21.175848 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:21.175596 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:06:33.188486 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.188442 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps"] Jul 09 22:06:33.191309 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.191289 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.194007 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.193985 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 09 22:06:33.201252 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.201229 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps"] Jul 09 22:06:33.339330 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.339292 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.339500 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.339345 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.339500 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.339388 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.339500 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.339403 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.339500 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.339486 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e57e32-00d3-4f03-83b0-9c151743559b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.339678 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.339515 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sspp\" (UniqueName: \"kubernetes.io/projected/c1e57e32-00d3-4f03-83b0-9c151743559b-kube-api-access-9sspp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.440481 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.440394 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e57e32-00d3-4f03-83b0-9c151743559b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.440481 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.440429 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sspp\" (UniqueName: \"kubernetes.io/projected/c1e57e32-00d3-4f03-83b0-9c151743559b-kube-api-access-9sspp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.440709 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.440487 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.440709 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.440513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.440709 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.440630 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.440709 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.440676 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.440943 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.440915 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.441058 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.440967 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.441058 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.441045 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.442742 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.442714 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1e57e32-00d3-4f03-83b0-9c151743559b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.443150 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.443130 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1e57e32-00d3-4f03-83b0-9c151743559b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.450000 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.449982 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sspp\" (UniqueName: \"kubernetes.io/projected/c1e57e32-00d3-4f03-83b0-9c151743559b-kube-api-access-9sspp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps\" (UID: \"c1e57e32-00d3-4f03-83b0-9c151743559b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.502746 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.502711 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:33.637544 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:33.637517 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps"] Jul 09 22:06:33.638877 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:06:33.638846 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e57e32_00d3_4f03_83b0_9c151743559b.slice/crio-75f46a2c77157f5bc050c7fd1a0814a619c80c233655c2f91152624c672cbe95 WatchSource:0}: Error finding container 75f46a2c77157f5bc050c7fd1a0814a619c80c233655c2f91152624c672cbe95: Status 404 returned error can't find the container with id 75f46a2c77157f5bc050c7fd1a0814a619c80c233655c2f91152624c672cbe95 Jul 09 22:06:34.428188 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:34.428155 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" event={"ID":"c1e57e32-00d3-4f03-83b0-9c151743559b","Type":"ContainerStarted","Data":"da1f07295aae688bd2cdceb80766c142f959f23985dce42c52d9394e18a6c1e0"} Jul 09 22:06:34.428669 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:34.428210 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" event={"ID":"c1e57e32-00d3-4f03-83b0-9c151743559b","Type":"ContainerStarted","Data":"75f46a2c77157f5bc050c7fd1a0814a619c80c233655c2f91152624c672cbe95"} Jul 09 22:06:34.448495 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:34.448445 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" podStartSLOduration=1.237435573 podStartE2EDuration="1.448427018s" totalImagesPullingTime="210.991445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:06:34.447690877 +0000 UTC m=+913.758308401" watchObservedRunningTime="2026-07-09 22:06:34.448427018 +0000 UTC m=+913.759044542" Jul 09 22:06:43.503457 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:43.503416 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:43.503893 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:43.503478 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:43.515990 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:43.515965 2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:06:44.493619 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:06:44.493589 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps" Jul 09 22:09:29.810734 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:29.810645 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-768b466dcc-jr4rl"] Jul 09 22:09:29.811260 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:29.810888 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-768b466dcc-jr4rl" podUID="25ca2676-c34a-4554-bc4a-0ed5ddf13e8c" containerName="manager" containerID="cri-o://eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6" gracePeriod=30 Jul 09 22:09:30.063827 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.063763 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:09:30.123278 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.123244 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbzrw\" (UniqueName: \"kubernetes.io/projected/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-kube-api-access-hbzrw\") pod \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\" (UID: \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\") " Jul 09 22:09:30.123437 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.123341 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-webhook-cert\") pod \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\" (UID: \"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c\") " Jul 09 22:09:30.125302 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.125267 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-kube-api-access-hbzrw" (OuterVolumeSpecName: "kube-api-access-hbzrw") pod "25ca2676-c34a-4554-bc4a-0ed5ddf13e8c" (UID: "25ca2676-c34a-4554-bc4a-0ed5ddf13e8c"). InnerVolumeSpecName "kube-api-access-hbzrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:09:30.125417 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.125351 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "25ca2676-c34a-4554-bc4a-0ed5ddf13e8c" (UID: "25ca2676-c34a-4554-bc4a-0ed5ddf13e8c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:09:30.224521 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.224487 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzrw\" (UniqueName: \"kubernetes.io/projected/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-kube-api-access-hbzrw\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:09:30.224521 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.224515 2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c-webhook-cert\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:09:30.379361 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.379327 2556 generic.go:356] "Generic (PLEG): container finished" podID="25ca2676-c34a-4554-bc4a-0ed5ddf13e8c" containerID="eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6" exitCode=0 Jul 09 22:09:30.379556 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.379374 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768b466dcc-jr4rl" Jul 09 22:09:30.379556 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.379412 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768b466dcc-jr4rl" event={"ID":"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c","Type":"ContainerDied","Data":"eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6"} Jul 09 22:09:30.379556 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.379468 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768b466dcc-jr4rl" event={"ID":"25ca2676-c34a-4554-bc4a-0ed5ddf13e8c","Type":"ContainerDied","Data":"3ab943d43e32000c0612dba7655e39ff4ea029fdde19db77fb324a967c48ceaa"} Jul 09 22:09:30.379556 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.379497 2556 scope.go:117] "RemoveContainer" containerID="eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6" Jul 09 22:09:30.389612 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.389565 2556 scope.go:117] "RemoveContainer" containerID="eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6" Jul 09 22:09:30.389838 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:09:30.389821 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6\": container with ID starting with eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6 not found: ID does not exist" containerID="eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6" Jul 09 22:09:30.389903 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.389844 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6"} err="failed to get container status \"eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6\": rpc error: code = NotFound desc = could not find container \"eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6\": container with ID starting with eadfe74cf30ea31cd5ba740995c18e42455592ba7ed2849da0d7a119f09d16d6 not found: ID does not exist" Jul 09 22:09:30.403652 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.403627 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-768b466dcc-jr4rl"] Jul 09 22:09:30.409740 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:30.409720 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-768b466dcc-jr4rl"] Jul 09 22:09:31.232651 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.232615 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-768b466dcc-vwgvf"] Jul 09 22:09:31.233250 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.233229 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ca2676-c34a-4554-bc4a-0ed5ddf13e8c" containerName="manager" Jul 09 22:09:31.233338 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.233252 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ca2676-c34a-4554-bc4a-0ed5ddf13e8c" containerName="manager" Jul 09 22:09:31.233391 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.233367 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ca2676-c34a-4554-bc4a-0ed5ddf13e8c" containerName="manager" Jul 09 22:09:31.236559 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.236537 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:31.240223 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.240199 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 22:09:31.240327 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.240199 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ddbdk\"" Jul 09 22:09:31.240327 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.240224 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 22:09:31.244859 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.244839 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768b466dcc-vwgvf"] Jul 09 22:09:31.276689 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.276659 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25ca2676-c34a-4554-bc4a-0ed5ddf13e8c" path="/var/lib/kubelet/pods/25ca2676-c34a-4554-bc4a-0ed5ddf13e8c/volumes" Jul 09 22:09:31.334405 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.334360 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5bk5\" (UniqueName: \"kubernetes.io/projected/ad82804c-00c4-4dc9-bcb3-2060628bebbc-kube-api-access-r5bk5\") pod \"maas-controller-768b466dcc-vwgvf\" (UID: \"ad82804c-00c4-4dc9-bcb3-2060628bebbc\") " pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:31.334627 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.334478 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad82804c-00c4-4dc9-bcb3-2060628bebbc-webhook-cert\") pod \"maas-controller-768b466dcc-vwgvf\" (UID: \"ad82804c-00c4-4dc9-bcb3-2060628bebbc\") " pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:31.435724 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.435677 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad82804c-00c4-4dc9-bcb3-2060628bebbc-webhook-cert\") pod \"maas-controller-768b466dcc-vwgvf\" (UID: \"ad82804c-00c4-4dc9-bcb3-2060628bebbc\") " pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:31.435930 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.435772 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5bk5\" (UniqueName: \"kubernetes.io/projected/ad82804c-00c4-4dc9-bcb3-2060628bebbc-kube-api-access-r5bk5\") pod \"maas-controller-768b466dcc-vwgvf\" (UID: \"ad82804c-00c4-4dc9-bcb3-2060628bebbc\") " pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:31.438342 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.438315 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad82804c-00c4-4dc9-bcb3-2060628bebbc-webhook-cert\") pod \"maas-controller-768b466dcc-vwgvf\" (UID: \"ad82804c-00c4-4dc9-bcb3-2060628bebbc\") " pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:31.444878 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.444857 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5bk5\" (UniqueName: \"kubernetes.io/projected/ad82804c-00c4-4dc9-bcb3-2060628bebbc-kube-api-access-r5bk5\") pod \"maas-controller-768b466dcc-vwgvf\" (UID: \"ad82804c-00c4-4dc9-bcb3-2060628bebbc\") " pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:31.548058 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.547955 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:31.680830 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.680803 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-768b466dcc-vwgvf"] Jul 09 22:09:31.681934 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:09:31.681891 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad82804c_00c4_4dc9_bcb3_2060628bebbc.slice/crio-6d690f33e76f1c553f5c0743ef09117286f76010b9007f401c08b35cd2142d95 WatchSource:0}: Error finding container 6d690f33e76f1c553f5c0743ef09117286f76010b9007f401c08b35cd2142d95: Status 404 returned error can't find the container with id 6d690f33e76f1c553f5c0743ef09117286f76010b9007f401c08b35cd2142d95 Jul 09 22:09:31.683175 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:31.683155 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:09:32.393453 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:32.393417 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768b466dcc-vwgvf" event={"ID":"ad82804c-00c4-4dc9-bcb3-2060628bebbc","Type":"ContainerStarted","Data":"6d690f33e76f1c553f5c0743ef09117286f76010b9007f401c08b35cd2142d95"} Jul 09 22:09:33.400457 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:33.400417 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-768b466dcc-vwgvf" event={"ID":"ad82804c-00c4-4dc9-bcb3-2060628bebbc","Type":"ContainerStarted","Data":"dfd58df1e147ccf2268ef286e50da68dd365bc58cba763f7ac3ffbb48f246981"} Jul 09 22:09:33.400933 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:33.400476 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:09:33.418778 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:33.418729 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-768b466dcc-vwgvf" podStartSLOduration=1.659606477 podStartE2EDuration="2.41871369s" totalImagesPullingTime="759.107213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:09:33.416803733 +0000 UTC m=+1092.727421256" watchObservedRunningTime="2026-07-09 22:09:33.41871369 +0000 UTC m=+1092.729331238" Jul 09 22:09:44.412167 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:09:44.412140 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-768b466dcc-vwgvf" Jul 09 22:11:21.219109 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:11:21.219078 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:11:21.221045 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:11:21.221023 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:16:21.256721 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:16:21.256666 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:16:21.258910 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:16:21.258888 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:21:21.304115 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:21.304090 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:21:21.307122 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:21.307100 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:21:25.978274 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:25.978241 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm"] Jul 09 22:21:25.981933 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:25.981911 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:25.984465 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:25.984444 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-61237e1b-serving-cert\"" Jul 09 22:21:25.999949 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:25.999924 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm"] Jul 09 22:21:26.070692 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.070629 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lztx\" (UniqueName: \"kubernetes.io/projected/0731816b-b6ce-42e8-bfe2-faad98bc2e00-kube-api-access-5lztx\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.070832 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.070717 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.070832 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.070760 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731816b-b6ce-42e8-bfe2-faad98bc2e00-openshift-service-ca\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.171522 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.171484 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731816b-b6ce-42e8-bfe2-faad98bc2e00-openshift-service-ca\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.171729 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.171639 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lztx\" (UniqueName: \"kubernetes.io/projected/0731816b-b6ce-42e8-bfe2-faad98bc2e00-kube-api-access-5lztx\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.171802 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.171734 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.172012 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:21:26.171985 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-serving-cert: secret "maas-api-e2e-ait-61237e1b-serving-cert" not found Jul 09 22:21:26.172283 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:21:26.172061 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls podName:0731816b-b6ce-42e8-bfe2-faad98bc2e00 nodeName:}" failed. No retries permitted until 2026-07-09 22:21:26.672040173 +0000 UTC m=+1805.982657678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls") pod "maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" (UID: "0731816b-b6ce-42e8-bfe2-faad98bc2e00") : secret "maas-api-e2e-ait-61237e1b-serving-cert" not found Jul 09 22:21:26.172516 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.172491 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731816b-b6ce-42e8-bfe2-faad98bc2e00-openshift-service-ca\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.180485 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.180461 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lztx\" (UniqueName: \"kubernetes.io/projected/0731816b-b6ce-42e8-bfe2-faad98bc2e00-kube-api-access-5lztx\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.676896 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.676861 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.679368 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.679339 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls\") pod \"maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:26.894007 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:26.893971 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:27.043882 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:27.043853 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm"] Jul 09 22:21:27.045668 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:21:27.045635 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0731816b_b6ce_42e8_bfe2_faad98bc2e00.slice/crio-783257fba2d82a61b6e1922a7b91b5fdf4cd7b3a9743984dd74cc97cf24ef519 WatchSource:0}: Error finding container 783257fba2d82a61b6e1922a7b91b5fdf4cd7b3a9743984dd74cc97cf24ef519: Status 404 returned error can't find the container with id 783257fba2d82a61b6e1922a7b91b5fdf4cd7b3a9743984dd74cc97cf24ef519 Jul 09 22:21:27.047661 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:27.047642 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:21:27.265234 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:27.265149 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" event={"ID":"0731816b-b6ce-42e8-bfe2-faad98bc2e00","Type":"ContainerStarted","Data":"783257fba2d82a61b6e1922a7b91b5fdf4cd7b3a9743984dd74cc97cf24ef519"} Jul 09 22:21:28.271267 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:28.271231 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" event={"ID":"0731816b-b6ce-42e8-bfe2-faad98bc2e00","Type":"ContainerStarted","Data":"ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86"} Jul 09 22:21:28.271670 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:28.271339 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:28.290289 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:28.290243 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" podStartSLOduration=2.755827333 podStartE2EDuration="3.290227805s" totalImagesPullingTime="534.400472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:21:28.288245026 +0000 UTC m=+1807.598862585" watchObservedRunningTime="2026-07-09 22:21:28.290227805 +0000 UTC m=+1807.600845329" Jul 09 22:21:31.387658 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.387624 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm"] Jul 09 22:21:31.388157 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.387798 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" podUID="0731816b-b6ce-42e8-bfe2-faad98bc2e00" containerName="maas-api" containerID="cri-o://ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86" gracePeriod=30 Jul 09 22:21:31.639358 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.639299 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:31.830329 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.830296 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls\") pod \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " Jul 09 22:21:31.830494 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.830350 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731816b-b6ce-42e8-bfe2-faad98bc2e00-openshift-service-ca\") pod \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " Jul 09 22:21:31.830543 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.830498 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lztx\" (UniqueName: \"kubernetes.io/projected/0731816b-b6ce-42e8-bfe2-faad98bc2e00-kube-api-access-5lztx\") pod \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\" (UID: \"0731816b-b6ce-42e8-bfe2-faad98bc2e00\") " Jul 09 22:21:31.830739 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.830719 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0731816b-b6ce-42e8-bfe2-faad98bc2e00-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0731816b-b6ce-42e8-bfe2-faad98bc2e00" (UID: "0731816b-b6ce-42e8-bfe2-faad98bc2e00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:21:31.830855 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.830834 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0731816b-b6ce-42e8-bfe2-faad98bc2e00-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:21:31.832659 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.832627 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0731816b-b6ce-42e8-bfe2-faad98bc2e00" (UID: "0731816b-b6ce-42e8-bfe2-faad98bc2e00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:21:31.832770 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.832734 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0731816b-b6ce-42e8-bfe2-faad98bc2e00-kube-api-access-5lztx" (OuterVolumeSpecName: "kube-api-access-5lztx") pod "0731816b-b6ce-42e8-bfe2-faad98bc2e00" (UID: "0731816b-b6ce-42e8-bfe2-faad98bc2e00"). InnerVolumeSpecName "kube-api-access-5lztx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:21:31.932343 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.932251 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lztx\" (UniqueName: \"kubernetes.io/projected/0731816b-b6ce-42e8-bfe2-faad98bc2e00-kube-api-access-5lztx\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:21:31.932343 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:31.932283 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0731816b-b6ce-42e8-bfe2-faad98bc2e00-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:21:32.297811 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.297727 2556 generic.go:356] "Generic (PLEG): container finished" podID="0731816b-b6ce-42e8-bfe2-faad98bc2e00" containerID="ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86" exitCode=0 Jul 09 22:21:32.297811 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.297769 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" Jul 09 22:21:32.297984 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.297816 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" event={"ID":"0731816b-b6ce-42e8-bfe2-faad98bc2e00","Type":"ContainerDied","Data":"ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86"} Jul 09 22:21:32.297984 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.297869 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm" event={"ID":"0731816b-b6ce-42e8-bfe2-faad98bc2e00","Type":"ContainerDied","Data":"783257fba2d82a61b6e1922a7b91b5fdf4cd7b3a9743984dd74cc97cf24ef519"} Jul 09 22:21:32.297984 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.297897 2556 scope.go:117] "RemoveContainer" containerID="ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86" Jul 09 22:21:32.307240 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.307220 2556 scope.go:117] "RemoveContainer" containerID="ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86" Jul 09 22:21:32.307509 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:21:32.307492 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86\": container with ID starting with ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86 not found: ID does not exist" containerID="ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86" Jul 09 22:21:32.307550 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.307519 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86"} err="failed to get container status \"ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86\": rpc error: code = NotFound desc = could not find container \"ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86\": container with ID starting with ba7e9e42c82a9470a989c97640ea1928f0da663c4cb2db4ffb387fd914f63a86 not found: ID does not exist" Jul 09 22:21:32.319370 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.319347 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm"] Jul 09 22:21:32.324049 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:32.324030 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-61237e1b-59c6447b7f-smnkm"] Jul 09 22:21:33.276193 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:33.276114 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0731816b-b6ce-42e8-bfe2-faad98bc2e00" path="/var/lib/kubelet/pods/0731816b-b6ce-42e8-bfe2-faad98bc2e00/volumes" Jul 09 22:21:38.597113 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.597074 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64"] Jul 09 22:21:38.597734 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.597714 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0731816b-b6ce-42e8-bfe2-faad98bc2e00" containerName="maas-api" Jul 09 22:21:38.597823 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.597738 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0731816b-b6ce-42e8-bfe2-faad98bc2e00" containerName="maas-api" Jul 09 22:21:38.597884 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.597869 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0731816b-b6ce-42e8-bfe2-faad98bc2e00" containerName="maas-api" Jul 09 22:21:38.602625 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.602604 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.605367 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.605345 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-80d0ec86-serving-cert\"" Jul 09 22:21:38.612356 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.612332 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64"] Jul 09 22:21:38.693791 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.693751 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9b4w\" (UniqueName: \"kubernetes.io/projected/0d2b775a-2312-49d0-b408-95fc0304656f-kube-api-access-g9b4w\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.693791 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.693796 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2b775a-2312-49d0-b408-95fc0304656f-openshift-service-ca\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.694089 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.693886 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2b775a-2312-49d0-b408-95fc0304656f-maas-api-tls\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.794463 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.794409 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9b4w\" (UniqueName: \"kubernetes.io/projected/0d2b775a-2312-49d0-b408-95fc0304656f-kube-api-access-g9b4w\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.794650 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.794472 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2b775a-2312-49d0-b408-95fc0304656f-openshift-service-ca\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.794650 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.794595 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2b775a-2312-49d0-b408-95fc0304656f-maas-api-tls\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.795295 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.795274 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2b775a-2312-49d0-b408-95fc0304656f-openshift-service-ca\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.797049 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.797031 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2b775a-2312-49d0-b408-95fc0304656f-maas-api-tls\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.805975 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.805947 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9b4w\" (UniqueName: \"kubernetes.io/projected/0d2b775a-2312-49d0-b408-95fc0304656f-kube-api-access-g9b4w\") pod \"maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:38.914367 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:38.914265 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:39.070971 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:39.070940 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64"] Jul 09 22:21:39.072693 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:21:39.072667 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2b775a_2312_49d0_b408_95fc0304656f.slice/crio-674f14773e9bf90a8a38b4963e6255db1eadacf64134b057ea8e4a9c47d77390 WatchSource:0}: Error finding container 674f14773e9bf90a8a38b4963e6255db1eadacf64134b057ea8e4a9c47d77390: Status 404 returned error can't find the container with id 674f14773e9bf90a8a38b4963e6255db1eadacf64134b057ea8e4a9c47d77390 Jul 09 22:21:39.338718 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:39.338687 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" event={"ID":"0d2b775a-2312-49d0-b408-95fc0304656f","Type":"ContainerStarted","Data":"674f14773e9bf90a8a38b4963e6255db1eadacf64134b057ea8e4a9c47d77390"} Jul 09 22:21:40.345617 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:40.345563 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" event={"ID":"0d2b775a-2312-49d0-b408-95fc0304656f","Type":"ContainerStarted","Data":"1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6"} Jul 09 22:21:40.346010 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:40.345773 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:40.363287 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:40.363230 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" podStartSLOduration=1.938720864 podStartE2EDuration="2.363216073s" totalImagesPullingTime="424.495209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:21:40.361697262 +0000 UTC m=+1819.672314785" watchObservedRunningTime="2026-07-09 22:21:40.363216073 +0000 UTC m=+1819.673833600" Jul 09 22:21:44.005157 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.005121 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64"] Jul 09 22:21:44.005646 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.005399 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" podUID="0d2b775a-2312-49d0-b408-95fc0304656f" containerName="maas-api" containerID="cri-o://1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6" gracePeriod=30 Jul 09 22:21:44.010788 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.010766 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:44.257893 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.257825 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:44.345943 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.345908 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9b4w\" (UniqueName: \"kubernetes.io/projected/0d2b775a-2312-49d0-b408-95fc0304656f-kube-api-access-g9b4w\") pod \"0d2b775a-2312-49d0-b408-95fc0304656f\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " Jul 09 22:21:44.346130 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.346001 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2b775a-2312-49d0-b408-95fc0304656f-openshift-service-ca\") pod \"0d2b775a-2312-49d0-b408-95fc0304656f\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " Jul 09 22:21:44.346130 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.346034 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2b775a-2312-49d0-b408-95fc0304656f-maas-api-tls\") pod \"0d2b775a-2312-49d0-b408-95fc0304656f\" (UID: \"0d2b775a-2312-49d0-b408-95fc0304656f\") " Jul 09 22:21:44.346387 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.346362 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2b775a-2312-49d0-b408-95fc0304656f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d2b775a-2312-49d0-b408-95fc0304656f" (UID: "0d2b775a-2312-49d0-b408-95fc0304656f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:21:44.348099 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.348070 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2b775a-2312-49d0-b408-95fc0304656f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d2b775a-2312-49d0-b408-95fc0304656f" (UID: "0d2b775a-2312-49d0-b408-95fc0304656f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:21:44.348204 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.348154 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2b775a-2312-49d0-b408-95fc0304656f-kube-api-access-g9b4w" (OuterVolumeSpecName: "kube-api-access-g9b4w") pod "0d2b775a-2312-49d0-b408-95fc0304656f" (UID: "0d2b775a-2312-49d0-b408-95fc0304656f"). InnerVolumeSpecName "kube-api-access-g9b4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:21:44.375925 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.375895 2556 generic.go:356] "Generic (PLEG): container finished" podID="0d2b775a-2312-49d0-b408-95fc0304656f" containerID="1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6" exitCode=0 Jul 09 22:21:44.376045 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.375938 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" Jul 09 22:21:44.376045 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.375984 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" event={"ID":"0d2b775a-2312-49d0-b408-95fc0304656f","Type":"ContainerDied","Data":"1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6"} Jul 09 22:21:44.376180 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.376045 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64" event={"ID":"0d2b775a-2312-49d0-b408-95fc0304656f","Type":"ContainerDied","Data":"674f14773e9bf90a8a38b4963e6255db1eadacf64134b057ea8e4a9c47d77390"} Jul 09 22:21:44.376180 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.376076 2556 scope.go:117] "RemoveContainer" containerID="1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6" Jul 09 22:21:44.388636 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.386467 2556 scope.go:117] "RemoveContainer" containerID="1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6" Jul 09 22:21:44.388636 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:21:44.388556 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6\": container with ID starting with 1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6 not found: ID does not exist" containerID="1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6" Jul 09 22:21:44.388636 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.388605 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6"} err="failed to get container status \"1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6\": rpc error: code = NotFound desc = could not find container \"1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6\": container with ID starting with 1abd58d403943185af3a0847a667b67ee07250a6fd87af6a18bbbbf1b16e17d6 not found: ID does not exist" Jul 09 22:21:44.401473 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.401448 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64"] Jul 09 22:21:44.406097 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.406075 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-80d0ec86-66f6c5fdf8-zcf64"] Jul 09 22:21:44.446889 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.446864 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2b775a-2312-49d0-b408-95fc0304656f-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:21:44.446889 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.446887 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d2b775a-2312-49d0-b408-95fc0304656f-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:21:44.447052 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:44.446897 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9b4w\" (UniqueName: \"kubernetes.io/projected/0d2b775a-2312-49d0-b408-95fc0304656f-kube-api-access-g9b4w\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:21:45.276070 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:45.276033 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d2b775a-2312-49d0-b408-95fc0304656f" path="/var/lib/kubelet/pods/0d2b775a-2312-49d0-b408-95fc0304656f/volumes" Jul 09 22:21:57.249990 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.249955 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w"] Jul 09 22:21:57.250459 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.250347 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2b775a-2312-49d0-b408-95fc0304656f" containerName="maas-api" Jul 09 22:21:57.250459 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.250358 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2b775a-2312-49d0-b408-95fc0304656f" containerName="maas-api" Jul 09 22:21:57.250459 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.250443 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d2b775a-2312-49d0-b408-95fc0304656f" containerName="maas-api" Jul 09 22:21:57.253389 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.253375 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.256255 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.256233 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-efc29080-serving-cert\"" Jul 09 22:21:57.264721 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.264699 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w"] Jul 09 22:21:57.368806 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.368770 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.368987 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.368837 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643e917d-310a-4660-bf4a-67ea39ecad3c-openshift-service-ca\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.368987 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.368943 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84jk\" (UniqueName: \"kubernetes.io/projected/643e917d-310a-4660-bf4a-67ea39ecad3c-kube-api-access-w84jk\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.469862 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.469828 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84jk\" (UniqueName: \"kubernetes.io/projected/643e917d-310a-4660-bf4a-67ea39ecad3c-kube-api-access-w84jk\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.470040 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.469905 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.470040 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.469976 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643e917d-310a-4660-bf4a-67ea39ecad3c-openshift-service-ca\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.470138 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:21:57.470060 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-serving-cert: secret "maas-api-e2e-derive-efc29080-serving-cert" not found Jul 09 22:21:57.470176 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:21:57.470146 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls podName:643e917d-310a-4660-bf4a-67ea39ecad3c nodeName:}" failed. No retries permitted until 2026-07-09 22:21:57.970126677 +0000 UTC m=+1837.280744177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls") pod "maas-api-e2e-derive-efc29080-8bf9f474-zp28w" (UID: "643e917d-310a-4660-bf4a-67ea39ecad3c") : secret "maas-api-e2e-derive-efc29080-serving-cert" not found Jul 09 22:21:57.470681 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.470664 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643e917d-310a-4660-bf4a-67ea39ecad3c-openshift-service-ca\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.482749 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.482724 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84jk\" (UniqueName: \"kubernetes.io/projected/643e917d-310a-4660-bf4a-67ea39ecad3c-kube-api-access-w84jk\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.975258 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.975225 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:57.977649 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:57.977631 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls\") pod \"maas-api-e2e-derive-efc29080-8bf9f474-zp28w\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:58.164759 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:58.164723 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:58.308378 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:58.308328 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w"] Jul 09 22:21:58.310251 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:21:58.310224 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643e917d_310a_4660_bf4a_67ea39ecad3c.slice/crio-14315459c09f813688ec8c72f34db372130a8458c49ed531a2a04402e66c1dac WatchSource:0}: Error finding container 14315459c09f813688ec8c72f34db372130a8458c49ed531a2a04402e66c1dac: Status 404 returned error can't find the container with id 14315459c09f813688ec8c72f34db372130a8458c49ed531a2a04402e66c1dac Jul 09 22:21:58.456188 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:58.456147 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" event={"ID":"643e917d-310a-4660-bf4a-67ea39ecad3c","Type":"ContainerStarted","Data":"14315459c09f813688ec8c72f34db372130a8458c49ed531a2a04402e66c1dac"} Jul 09 22:21:59.462610 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:59.462558 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" event={"ID":"643e917d-310a-4660-bf4a-67ea39ecad3c","Type":"ContainerStarted","Data":"e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47"} Jul 09 22:21:59.462987 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:59.462747 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:21:59.480492 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:21:59.480446 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" podStartSLOduration=2.120364941 podStartE2EDuration="2.480432895s" totalImagesPullingTime="360.067954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:21:59.479430381 +0000 UTC m=+1838.790047896" watchObservedRunningTime="2026-07-09 22:21:59.480432895 +0000 UTC m=+1838.791050418" Jul 09 22:22:01.928497 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:01.928463 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w"] Jul 09 22:22:01.928966 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:01.928729 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" podUID="643e917d-310a-4660-bf4a-67ea39ecad3c" containerName="maas-api" containerID="cri-o://e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47" gracePeriod=30 Jul 09 22:22:02.174946 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.174923 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:22:02.316976 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.316893 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls\") pod \"643e917d-310a-4660-bf4a-67ea39ecad3c\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " Jul 09 22:22:02.316976 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.316933 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643e917d-310a-4660-bf4a-67ea39ecad3c-openshift-service-ca\") pod \"643e917d-310a-4660-bf4a-67ea39ecad3c\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " Jul 09 22:22:02.317187 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.317072 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w84jk\" (UniqueName: \"kubernetes.io/projected/643e917d-310a-4660-bf4a-67ea39ecad3c-kube-api-access-w84jk\") pod \"643e917d-310a-4660-bf4a-67ea39ecad3c\" (UID: \"643e917d-310a-4660-bf4a-67ea39ecad3c\") " Jul 09 22:22:02.317341 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.317318 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643e917d-310a-4660-bf4a-67ea39ecad3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "643e917d-310a-4660-bf4a-67ea39ecad3c" (UID: "643e917d-310a-4660-bf4a-67ea39ecad3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:22:02.319139 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.319113 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "643e917d-310a-4660-bf4a-67ea39ecad3c" (UID: "643e917d-310a-4660-bf4a-67ea39ecad3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:22:02.319241 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.319119 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643e917d-310a-4660-bf4a-67ea39ecad3c-kube-api-access-w84jk" (OuterVolumeSpecName: "kube-api-access-w84jk") pod "643e917d-310a-4660-bf4a-67ea39ecad3c" (UID: "643e917d-310a-4660-bf4a-67ea39ecad3c"). InnerVolumeSpecName "kube-api-access-w84jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:22:02.418104 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.418074 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/643e917d-310a-4660-bf4a-67ea39ecad3c-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:22:02.418104 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.418101 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/643e917d-310a-4660-bf4a-67ea39ecad3c-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:22:02.418104 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.418112 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w84jk\" (UniqueName: \"kubernetes.io/projected/643e917d-310a-4660-bf4a-67ea39ecad3c-kube-api-access-w84jk\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:22:02.482373 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.482343 2556 generic.go:356] "Generic (PLEG): container finished" podID="643e917d-310a-4660-bf4a-67ea39ecad3c" containerID="e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47" exitCode=0 Jul 09 22:22:02.482545 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.482393 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" Jul 09 22:22:02.482545 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.482395 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" event={"ID":"643e917d-310a-4660-bf4a-67ea39ecad3c","Type":"ContainerDied","Data":"e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47"} Jul 09 22:22:02.482545 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.482523 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w" event={"ID":"643e917d-310a-4660-bf4a-67ea39ecad3c","Type":"ContainerDied","Data":"14315459c09f813688ec8c72f34db372130a8458c49ed531a2a04402e66c1dac"} Jul 09 22:22:02.482750 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.482552 2556 scope.go:117] "RemoveContainer" containerID="e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47" Jul 09 22:22:02.492431 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.492416 2556 scope.go:117] "RemoveContainer" containerID="e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47" Jul 09 22:22:02.492721 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:22:02.492703 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47\": container with ID starting with e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47 not found: ID does not exist" containerID="e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47" Jul 09 22:22:02.492777 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.492729 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47"} err="failed to get container status \"e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47\": rpc error: code = NotFound desc = could not find container \"e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47\": container with ID starting with e8486d96f78513a3b8064b660fdb0562adea571f5d327dabaac6e2c2f9e54a47 not found: ID does not exist" Jul 09 22:22:02.507464 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.507437 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w"] Jul 09 22:22:02.513319 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:02.513296 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-efc29080-8bf9f474-zp28w"] Jul 09 22:22:03.277940 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:22:03.277903 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643e917d-310a-4660-bf4a-67ea39ecad3c" path="/var/lib/kubelet/pods/643e917d-310a-4660-bf4a-67ea39ecad3c/volumes" Jul 09 22:23:46.857533 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.857498 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn"] Jul 09 22:23:46.858171 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.857948 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643e917d-310a-4660-bf4a-67ea39ecad3c" containerName="maas-api" Jul 09 22:23:46.858171 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.857961 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="643e917d-310a-4660-bf4a-67ea39ecad3c" containerName="maas-api" Jul 09 22:23:46.858171 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.858027 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="643e917d-310a-4660-bf4a-67ea39ecad3c" containerName="maas-api" Jul 09 22:23:46.861135 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.861116 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:46.864714 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.864689 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8389e858-serving-cert\"" Jul 09 22:23:46.875977 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.875951 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn"] Jul 09 22:23:46.987283 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.987244 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01f25a9-ea5a-43ed-b98e-8a8c52432038-openshift-service-ca\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:46.987459 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.987312 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhj2v\" (UniqueName: \"kubernetes.io/projected/c01f25a9-ea5a-43ed-b98e-8a8c52432038-kube-api-access-nhj2v\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:46.987459 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:46.987353 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01f25a9-ea5a-43ed-b98e-8a8c52432038-maas-api-tls\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:47.088761 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:47.088728 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01f25a9-ea5a-43ed-b98e-8a8c52432038-openshift-service-ca\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:47.088945 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:47.088796 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhj2v\" (UniqueName: \"kubernetes.io/projected/c01f25a9-ea5a-43ed-b98e-8a8c52432038-kube-api-access-nhj2v\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:47.088945 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:47.088826 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01f25a9-ea5a-43ed-b98e-8a8c52432038-maas-api-tls\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:47.089408 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:47.089367 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01f25a9-ea5a-43ed-b98e-8a8c52432038-openshift-service-ca\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:47.091268 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:47.091247 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01f25a9-ea5a-43ed-b98e-8a8c52432038-maas-api-tls\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:47.146568 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:47.146502 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhj2v\" (UniqueName: \"kubernetes.io/projected/c01f25a9-ea5a-43ed-b98e-8a8c52432038-kube-api-access-nhj2v\") pod \"maas-api-e2e-mt-8389e858-585cb7868c-srggn\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:47.171344 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:47.171316 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:47.330600 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:47.330558 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn"] Jul 09 22:23:47.331812 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:23:47.331783 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01f25a9_ea5a_43ed_b98e_8a8c52432038.slice/crio-5d7771898706ef7d6ef86eec5d548ccaf2ef5831f7cdb9337e63c5f95210c3f0 WatchSource:0}: Error finding container 5d7771898706ef7d6ef86eec5d548ccaf2ef5831f7cdb9337e63c5f95210c3f0: Status 404 returned error can't find the container with id 5d7771898706ef7d6ef86eec5d548ccaf2ef5831f7cdb9337e63c5f95210c3f0 Jul 09 22:23:48.042558 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:48.042468 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" event={"ID":"c01f25a9-ea5a-43ed-b98e-8a8c52432038","Type":"ContainerStarted","Data":"ee8bc968e51963d33696634d5c6954c77abcf8e911859613eef79b6fadbfa4c6"} Jul 09 22:23:48.042558 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:48.042541 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:48.042974 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:48.042565 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" event={"ID":"c01f25a9-ea5a-43ed-b98e-8a8c52432038","Type":"ContainerStarted","Data":"5d7771898706ef7d6ef86eec5d548ccaf2ef5831f7cdb9337e63c5f95210c3f0"} Jul 09 22:23:48.066669 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:48.066605 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" podStartSLOduration=1.628989515 podStartE2EDuration="2.066570048s" totalImagesPullingTime="437.580533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:23:48.065280883 +0000 UTC m=+1947.375898407" watchObservedRunningTime="2026-07-09 22:23:48.066570048 +0000 UTC m=+1947.377187571" Jul 09 22:23:54.055429 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:54.055385 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:58.977501 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:58.977470 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn"] Jul 09 22:23:58.977961 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:58.977742 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" podUID="c01f25a9-ea5a-43ed-b98e-8a8c52432038" containerName="maas-api" containerID="cri-o://ee8bc968e51963d33696634d5c6954c77abcf8e911859613eef79b6fadbfa4c6" gracePeriod=30 Jul 09 22:23:59.049411 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.049379 2556 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" podUID="c01f25a9-ea5a-43ed-b98e-8a8c52432038" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused" Jul 09 22:23:59.105506 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.105473 2556 generic.go:356] "Generic (PLEG): container finished" podID="c01f25a9-ea5a-43ed-b98e-8a8c52432038" containerID="ee8bc968e51963d33696634d5c6954c77abcf8e911859613eef79b6fadbfa4c6" exitCode=0 Jul 09 22:23:59.105661 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.105541 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" event={"ID":"c01f25a9-ea5a-43ed-b98e-8a8c52432038","Type":"ContainerDied","Data":"ee8bc968e51963d33696634d5c6954c77abcf8e911859613eef79b6fadbfa4c6"} Jul 09 22:23:59.226801 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.226779 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:23:59.293659 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.293542 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01f25a9-ea5a-43ed-b98e-8a8c52432038-maas-api-tls\") pod \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " Jul 09 22:23:59.293659 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.293633 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01f25a9-ea5a-43ed-b98e-8a8c52432038-openshift-service-ca\") pod \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " Jul 09 22:23:59.293882 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.293698 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhj2v\" (UniqueName: \"kubernetes.io/projected/c01f25a9-ea5a-43ed-b98e-8a8c52432038-kube-api-access-nhj2v\") pod \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\" (UID: \"c01f25a9-ea5a-43ed-b98e-8a8c52432038\") " Jul 09 22:23:59.294006 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.293972 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01f25a9-ea5a-43ed-b98e-8a8c52432038-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c01f25a9-ea5a-43ed-b98e-8a8c52432038" (UID: "c01f25a9-ea5a-43ed-b98e-8a8c52432038"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:23:59.295634 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.295602 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01f25a9-ea5a-43ed-b98e-8a8c52432038-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c01f25a9-ea5a-43ed-b98e-8a8c52432038" (UID: "c01f25a9-ea5a-43ed-b98e-8a8c52432038"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:23:59.295724 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.295632 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01f25a9-ea5a-43ed-b98e-8a8c52432038-kube-api-access-nhj2v" (OuterVolumeSpecName: "kube-api-access-nhj2v") pod "c01f25a9-ea5a-43ed-b98e-8a8c52432038" (UID: "c01f25a9-ea5a-43ed-b98e-8a8c52432038"). InnerVolumeSpecName "kube-api-access-nhj2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:23:59.394840 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.394803 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhj2v\" (UniqueName: \"kubernetes.io/projected/c01f25a9-ea5a-43ed-b98e-8a8c52432038-kube-api-access-nhj2v\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:23:59.394840 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.394834 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c01f25a9-ea5a-43ed-b98e-8a8c52432038-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:23:59.394840 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:23:59.394843 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c01f25a9-ea5a-43ed-b98e-8a8c52432038-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:24:00.112079 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:00.112051 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" Jul 09 22:24:00.112516 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:00.112090 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn" event={"ID":"c01f25a9-ea5a-43ed-b98e-8a8c52432038","Type":"ContainerDied","Data":"5d7771898706ef7d6ef86eec5d548ccaf2ef5831f7cdb9337e63c5f95210c3f0"} Jul 09 22:24:00.112516 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:00.112151 2556 scope.go:117] "RemoveContainer" containerID="ee8bc968e51963d33696634d5c6954c77abcf8e911859613eef79b6fadbfa4c6" Jul 09 22:24:00.137015 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:00.136985 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn"] Jul 09 22:24:00.140283 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:00.140249 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8389e858-585cb7868c-srggn"] Jul 09 22:24:01.276342 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:01.276311 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01f25a9-ea5a-43ed-b98e-8a8c52432038" path="/var/lib/kubelet/pods/c01f25a9-ea5a-43ed-b98e-8a8c52432038/volumes" Jul 09 22:24:28.244695 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.244655 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6"] Jul 09 22:24:28.245124 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.245068 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01f25a9-ea5a-43ed-b98e-8a8c52432038" containerName="maas-api" Jul 09 22:24:28.245124 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.245080 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01f25a9-ea5a-43ed-b98e-8a8c52432038" containerName="maas-api" Jul 09 22:24:28.245205 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.245160 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01f25a9-ea5a-43ed-b98e-8a8c52432038" containerName="maas-api" Jul 09 22:24:28.248264 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.248242 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.250831 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.250810 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f4b46590-serving-cert\"" Jul 09 22:24:28.259467 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.259434 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6"] Jul 09 22:24:28.361430 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.361398 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.361617 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.361474 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97f8aec-5ef9-4631-bae1-608b52fa50db-openshift-service-ca\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.361617 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.361510 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjp4h\" (UniqueName: \"kubernetes.io/projected/d97f8aec-5ef9-4631-bae1-608b52fa50db-kube-api-access-bjp4h\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.462439 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.462404 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.462658 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.462476 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97f8aec-5ef9-4631-bae1-608b52fa50db-openshift-service-ca\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.462658 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.462513 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp4h\" (UniqueName: \"kubernetes.io/projected/d97f8aec-5ef9-4631-bae1-608b52fa50db-kube-api-access-bjp4h\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.462658 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:24:28.462554 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-serving-cert: secret "maas-api-e2e-mt-f4b46590-serving-cert" not found Jul 09 22:24:28.462658 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:24:28.462653 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls podName:d97f8aec-5ef9-4631-bae1-608b52fa50db nodeName:}" failed. No retries permitted until 2026-07-09 22:24:28.962634323 +0000 UTC m=+1988.273251824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls") pod "maas-api-e2e-mt-f4b46590-7f6f478885-269j6" (UID: "d97f8aec-5ef9-4631-bae1-608b52fa50db") : secret "maas-api-e2e-mt-f4b46590-serving-cert" not found Jul 09 22:24:28.463096 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.463078 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97f8aec-5ef9-4631-bae1-608b52fa50db-openshift-service-ca\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.471256 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.471234 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp4h\" (UniqueName: \"kubernetes.io/projected/d97f8aec-5ef9-4631-bae1-608b52fa50db-kube-api-access-bjp4h\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.967768 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.967731 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:28.970153 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:28.970123 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls\") pod \"maas-api-e2e-mt-f4b46590-7f6f478885-269j6\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:29.160744 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:29.160707 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:29.297788 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:29.297765 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6"] Jul 09 22:24:29.299688 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:24:29.299664 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97f8aec_5ef9_4631_bae1_608b52fa50db.slice/crio-31a3f75abcfd9bfa9db3ea6ef2cc308cd5613903f3f2d6f62a2ba15a26e232ba WatchSource:0}: Error finding container 31a3f75abcfd9bfa9db3ea6ef2cc308cd5613903f3f2d6f62a2ba15a26e232ba: Status 404 returned error can't find the container with id 31a3f75abcfd9bfa9db3ea6ef2cc308cd5613903f3f2d6f62a2ba15a26e232ba Jul 09 22:24:30.279225 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:30.279132 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" event={"ID":"d97f8aec-5ef9-4631-bae1-608b52fa50db","Type":"ContainerStarted","Data":"476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96"} Jul 09 22:24:30.279225 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:30.279191 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" event={"ID":"d97f8aec-5ef9-4631-bae1-608b52fa50db","Type":"ContainerStarted","Data":"31a3f75abcfd9bfa9db3ea6ef2cc308cd5613903f3f2d6f62a2ba15a26e232ba"} Jul 09 22:24:30.279419 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:30.279278 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:24:30.300007 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:30.299955 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" podStartSLOduration=1.746668278 podStartE2EDuration="2.299941265s" totalImagesPullingTime="553.272987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:24:30.297955726 +0000 UTC m=+1989.608573282" watchObservedRunningTime="2026-07-09 22:24:30.299941265 +0000 UTC m=+1989.610558789" Jul 09 22:24:36.290971 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:24:36.290943 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:25:15.918431 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:15.918396 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6"] Jul 09 22:25:15.918952 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:15.918633 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" podUID="d97f8aec-5ef9-4631-bae1-608b52fa50db" containerName="maas-api" containerID="cri-o://476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96" gracePeriod=30 Jul 09 22:25:16.165274 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.165250 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:25:16.218999 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.218912 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97f8aec-5ef9-4631-bae1-608b52fa50db-openshift-service-ca\") pod \"d97f8aec-5ef9-4631-bae1-608b52fa50db\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " Jul 09 22:25:16.219314 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.219290 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97f8aec-5ef9-4631-bae1-608b52fa50db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d97f8aec-5ef9-4631-bae1-608b52fa50db" (UID: "d97f8aec-5ef9-4631-bae1-608b52fa50db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:25:16.320054 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.320022 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls\") pod \"d97f8aec-5ef9-4631-bae1-608b52fa50db\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " Jul 09 22:25:16.320218 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.320064 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjp4h\" (UniqueName: \"kubernetes.io/projected/d97f8aec-5ef9-4631-bae1-608b52fa50db-kube-api-access-bjp4h\") pod \"d97f8aec-5ef9-4631-bae1-608b52fa50db\" (UID: \"d97f8aec-5ef9-4631-bae1-608b52fa50db\") " Jul 09 22:25:16.320353 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.320335 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d97f8aec-5ef9-4631-bae1-608b52fa50db-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:25:16.322225 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.322200 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d97f8aec-5ef9-4631-bae1-608b52fa50db" (UID: "d97f8aec-5ef9-4631-bae1-608b52fa50db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:25:16.322318 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.322231 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97f8aec-5ef9-4631-bae1-608b52fa50db-kube-api-access-bjp4h" (OuterVolumeSpecName: "kube-api-access-bjp4h") pod "d97f8aec-5ef9-4631-bae1-608b52fa50db" (UID: "d97f8aec-5ef9-4631-bae1-608b52fa50db"). InnerVolumeSpecName "kube-api-access-bjp4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:25:16.421667 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.421638 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjp4h\" (UniqueName: \"kubernetes.io/projected/d97f8aec-5ef9-4631-bae1-608b52fa50db-kube-api-access-bjp4h\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:25:16.421667 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.421665 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d97f8aec-5ef9-4631-bae1-608b52fa50db-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:25:16.525582 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.525484 2556 generic.go:356] "Generic (PLEG): container finished" podID="d97f8aec-5ef9-4631-bae1-608b52fa50db" containerID="476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96" exitCode=0 Jul 09 22:25:16.525582 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.525528 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" Jul 09 22:25:16.525775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.525586 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" event={"ID":"d97f8aec-5ef9-4631-bae1-608b52fa50db","Type":"ContainerDied","Data":"476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96"} Jul 09 22:25:16.525775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.525647 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6" event={"ID":"d97f8aec-5ef9-4631-bae1-608b52fa50db","Type":"ContainerDied","Data":"31a3f75abcfd9bfa9db3ea6ef2cc308cd5613903f3f2d6f62a2ba15a26e232ba"} Jul 09 22:25:16.525775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.525675 2556 scope.go:117] "RemoveContainer" containerID="476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96" Jul 09 22:25:16.535422 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.535406 2556 scope.go:117] "RemoveContainer" containerID="476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96" Jul 09 22:25:16.535703 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:25:16.535680 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96\": container with ID starting with 476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96 not found: ID does not exist" containerID="476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96" Jul 09 22:25:16.535767 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.535711 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96"} err="failed to get container status \"476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96\": rpc error: code = NotFound desc = could not find container \"476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96\": container with ID starting with 476931e92dba918c06d3e89b23f74dddb5a0a6dc5676043bb6b7f28f3c9c5d96 not found: ID does not exist" Jul 09 22:25:16.550149 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.550125 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6"] Jul 09 22:25:16.554926 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:16.554904 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f4b46590-7f6f478885-269j6"] Jul 09 22:25:17.276066 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:25:17.276036 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97f8aec-5ef9-4631-bae1-608b52fa50db" path="/var/lib/kubelet/pods/d97f8aec-5ef9-4631-bae1-608b52fa50db/volumes" Jul 09 22:26:21.340812 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:21.340780 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:26:21.345722 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:21.345700 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:26:52.654127 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.654089 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl"] Jul 09 22:26:52.654590 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.654485 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97f8aec-5ef9-4631-bae1-608b52fa50db" containerName="maas-api" Jul 09 22:26:52.654590 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.654497 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97f8aec-5ef9-4631-bae1-608b52fa50db" containerName="maas-api" Jul 09 22:26:52.654590 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.654569 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97f8aec-5ef9-4631-bae1-608b52fa50db" containerName="maas-api" Jul 09 22:26:52.657723 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.657703 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:52.660061 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.660039 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-55e48b-serving-cert\"" Jul 09 22:26:52.668850 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.668824 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl"] Jul 09 22:26:52.717445 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.717416 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae866552-4a1d-4987-b68d-93668d647e11-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:52.717611 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.717481 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:52.717611 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.717549 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwnh\" (UniqueName: \"kubernetes.io/projected/ae866552-4a1d-4987-b68d-93668d647e11-kube-api-access-blwnh\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:52.818972 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.818928 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwnh\" (UniqueName: \"kubernetes.io/projected/ae866552-4a1d-4987-b68d-93668d647e11-kube-api-access-blwnh\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:52.819180 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.819018 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae866552-4a1d-4987-b68d-93668d647e11-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:52.819180 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.819105 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:52.819310 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:26:52.819232 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-serving-cert: secret "maas-api-e2e-inf-a-55e48b-serving-cert" not found Jul 09 22:26:52.819365 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:26:52.819329 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls podName:ae866552-4a1d-4987-b68d-93668d647e11 nodeName:}" failed. No retries permitted until 2026-07-09 22:26:53.319307042 +0000 UTC m=+2132.629924543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls") pod "maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" (UID: "ae866552-4a1d-4987-b68d-93668d647e11") : secret "maas-api-e2e-inf-a-55e48b-serving-cert" not found Jul 09 22:26:52.819660 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.819639 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae866552-4a1d-4987-b68d-93668d647e11-openshift-service-ca\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:52.827761 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:52.827735 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwnh\" (UniqueName: \"kubernetes.io/projected/ae866552-4a1d-4987-b68d-93668d647e11-kube-api-access-blwnh\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:53.324462 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:53.324422 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:53.326969 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:53.326937 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls\") pod \"maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:53.569300 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:53.569212 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:53.712689 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:53.712648 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl"] Jul 09 22:26:53.714806 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:26:53.714778 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae866552_4a1d_4987_b68d_93668d647e11.slice/crio-b9f05245692be60e1913f1b3db81c825638676a326b15a115e97adb2e23a6468 WatchSource:0}: Error finding container b9f05245692be60e1913f1b3db81c825638676a326b15a115e97adb2e23a6468: Status 404 returned error can't find the container with id b9f05245692be60e1913f1b3db81c825638676a326b15a115e97adb2e23a6468 Jul 09 22:26:53.716821 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:53.716801 2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:26:54.043252 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:54.043215 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" event={"ID":"ae866552-4a1d-4987-b68d-93668d647e11","Type":"ContainerStarted","Data":"b9f05245692be60e1913f1b3db81c825638676a326b15a115e97adb2e23a6468"} Jul 09 22:26:55.050358 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:55.050324 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" event={"ID":"ae866552-4a1d-4987-b68d-93668d647e11","Type":"ContainerStarted","Data":"22cff41141d92f3c9cac083c6063d60f86442e0b499d5d850dec5eccb1604c3e"} Jul 09 22:26:55.050768 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:55.050382 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:26:55.073868 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:26:55.073813 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" podStartSLOduration=2.410149132 podStartE2EDuration="3.073796925s" totalImagesPullingTime="663.647793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:26:55.072238775 +0000 UTC m=+2134.382856302" watchObservedRunningTime="2026-07-09 22:26:55.073796925 +0000 UTC m=+2134.384414448" Jul 09 22:27:00.723393 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.723359 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h"] Jul 09 22:27:00.727410 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.727382 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:00.730261 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.730240 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-03e1d1-serving-cert\"" Jul 09 22:27:00.738659 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.738637 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h"] Jul 09 22:27:00.794693 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.794654 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da9f181-55c2-40cd-ba0c-61781f00784d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:00.794867 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.794710 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:00.794867 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.794733 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzbc5\" (UniqueName: \"kubernetes.io/projected/1da9f181-55c2-40cd-ba0c-61781f00784d-kube-api-access-pzbc5\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:00.895639 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.895602 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da9f181-55c2-40cd-ba0c-61781f00784d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:00.895838 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.895664 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:00.895838 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.895698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzbc5\" (UniqueName: \"kubernetes.io/projected/1da9f181-55c2-40cd-ba0c-61781f00784d-kube-api-access-pzbc5\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:00.895838 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:27:00.895826 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-serving-cert: secret "maas-api-e2e-inf-b-03e1d1-serving-cert" not found Jul 09 22:27:00.896008 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:27:00.895890 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls podName:1da9f181-55c2-40cd-ba0c-61781f00784d nodeName:}" failed. No retries permitted until 2026-07-09 22:27:01.395873981 +0000 UTC m=+2140.706491481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls") pod "maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" (UID: "1da9f181-55c2-40cd-ba0c-61781f00784d") : secret "maas-api-e2e-inf-b-03e1d1-serving-cert" not found Jul 09 22:27:00.896264 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.896246 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da9f181-55c2-40cd-ba0c-61781f00784d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:00.906635 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:00.906614 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzbc5\" (UniqueName: \"kubernetes.io/projected/1da9f181-55c2-40cd-ba0c-61781f00784d-kube-api-access-pzbc5\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:01.061458 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:01.061376 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:27:01.400196 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:01.400162 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:01.402623 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:01.402597 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls\") pod \"maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:01.639156 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:01.639124 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:01.780303 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:01.780275 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h"] Jul 09 22:27:01.782108 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:27:01.782080 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da9f181_55c2_40cd_ba0c_61781f00784d.slice/crio-f89c9538c0bedd6869031371bb6a7157ccee641bb4abee7d3a4ff7d79706a041 WatchSource:0}: Error finding container f89c9538c0bedd6869031371bb6a7157ccee641bb4abee7d3a4ff7d79706a041: Status 404 returned error can't find the container with id f89c9538c0bedd6869031371bb6a7157ccee641bb4abee7d3a4ff7d79706a041 Jul 09 22:27:02.093819 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:02.093782 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" event={"ID":"1da9f181-55c2-40cd-ba0c-61781f00784d","Type":"ContainerStarted","Data":"f89c9538c0bedd6869031371bb6a7157ccee641bb4abee7d3a4ff7d79706a041"} Jul 09 22:27:03.101334 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:03.101297 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" event={"ID":"1da9f181-55c2-40cd-ba0c-61781f00784d","Type":"ContainerStarted","Data":"bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046"} Jul 09 22:27:03.101812 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:03.101366 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:27:03.121845 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:03.121795 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" podStartSLOduration=2.657655995 podStartE2EDuration="3.121781526s" totalImagesPullingTime="464.125531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:27:03.120694238 +0000 UTC m=+2142.431311764" watchObservedRunningTime="2026-07-09 22:27:03.121781526 +0000 UTC m=+2142.432399048" Jul 09 22:27:09.113324 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:27:09.113294 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:28:26.360264 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.360219 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl"] Jul 09 22:28:26.360905 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.360486 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" podUID="ae866552-4a1d-4987-b68d-93668d647e11" containerName="maas-api" containerID="cri-o://22cff41141d92f3c9cac083c6063d60f86442e0b499d5d850dec5eccb1604c3e" gracePeriod=30 Jul 09 22:28:26.561792 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.561759 2556 generic.go:356] "Generic (PLEG): container finished" podID="ae866552-4a1d-4987-b68d-93668d647e11" containerID="22cff41141d92f3c9cac083c6063d60f86442e0b499d5d850dec5eccb1604c3e" exitCode=0 Jul 09 22:28:26.561967 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.561802 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" event={"ID":"ae866552-4a1d-4987-b68d-93668d647e11","Type":"ContainerDied","Data":"22cff41141d92f3c9cac083c6063d60f86442e0b499d5d850dec5eccb1604c3e"} Jul 09 22:28:26.617847 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.617788 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:28:26.719797 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.719763 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls\") pod \"ae866552-4a1d-4987-b68d-93668d647e11\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " Jul 09 22:28:26.720005 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.719823 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae866552-4a1d-4987-b68d-93668d647e11-openshift-service-ca\") pod \"ae866552-4a1d-4987-b68d-93668d647e11\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " Jul 09 22:28:26.720005 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.719848 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blwnh\" (UniqueName: \"kubernetes.io/projected/ae866552-4a1d-4987-b68d-93668d647e11-kube-api-access-blwnh\") pod \"ae866552-4a1d-4987-b68d-93668d647e11\" (UID: \"ae866552-4a1d-4987-b68d-93668d647e11\") " Jul 09 22:28:26.720237 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.720211 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae866552-4a1d-4987-b68d-93668d647e11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae866552-4a1d-4987-b68d-93668d647e11" (UID: "ae866552-4a1d-4987-b68d-93668d647e11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:28:26.721990 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.721966 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae866552-4a1d-4987-b68d-93668d647e11" (UID: "ae866552-4a1d-4987-b68d-93668d647e11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:28:26.722079 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.722060 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae866552-4a1d-4987-b68d-93668d647e11-kube-api-access-blwnh" (OuterVolumeSpecName: "kube-api-access-blwnh") pod "ae866552-4a1d-4987-b68d-93668d647e11" (UID: "ae866552-4a1d-4987-b68d-93668d647e11"). InnerVolumeSpecName "kube-api-access-blwnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:28:26.820556 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.820521 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae866552-4a1d-4987-b68d-93668d647e11-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:28:26.820556 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.820554 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae866552-4a1d-4987-b68d-93668d647e11-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:28:26.820556 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:26.820564 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blwnh\" (UniqueName: \"kubernetes.io/projected/ae866552-4a1d-4987-b68d-93668d647e11-kube-api-access-blwnh\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:28:27.568929 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:27.568844 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" Jul 09 22:28:27.568929 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:27.568845 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl" event={"ID":"ae866552-4a1d-4987-b68d-93668d647e11","Type":"ContainerDied","Data":"b9f05245692be60e1913f1b3db81c825638676a326b15a115e97adb2e23a6468"} Jul 09 22:28:27.569421 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:27.568920 2556 scope.go:117] "RemoveContainer" containerID="22cff41141d92f3c9cac083c6063d60f86442e0b499d5d850dec5eccb1604c3e" Jul 09 22:28:27.590773 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:27.590746 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl"] Jul 09 22:28:27.595142 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:27.595117 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-55e48b-6dbd45d8d4-kg5zl"] Jul 09 22:28:29.277379 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:28:29.277345 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae866552-4a1d-4987-b68d-93668d647e11" path="/var/lib/kubelet/pods/ae866552-4a1d-4987-b68d-93668d647e11/volumes" Jul 09 22:29:10.125417 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.125377 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h"] Jul 09 22:29:10.126080 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.125643 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" podUID="1da9f181-55c2-40cd-ba0c-61781f00784d" containerName="maas-api" containerID="cri-o://bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046" gracePeriod=30 Jul 09 22:29:10.576913 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.576886 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:29:10.744069 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.743985 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzbc5\" (UniqueName: \"kubernetes.io/projected/1da9f181-55c2-40cd-ba0c-61781f00784d-kube-api-access-pzbc5\") pod \"1da9f181-55c2-40cd-ba0c-61781f00784d\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " Jul 09 22:29:10.744069 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.744025 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da9f181-55c2-40cd-ba0c-61781f00784d-openshift-service-ca\") pod \"1da9f181-55c2-40cd-ba0c-61781f00784d\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " Jul 09 22:29:10.744251 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.744123 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls\") pod \"1da9f181-55c2-40cd-ba0c-61781f00784d\" (UID: \"1da9f181-55c2-40cd-ba0c-61781f00784d\") " Jul 09 22:29:10.744386 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.744365 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da9f181-55c2-40cd-ba0c-61781f00784d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1da9f181-55c2-40cd-ba0c-61781f00784d" (UID: "1da9f181-55c2-40cd-ba0c-61781f00784d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:29:10.744464 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.744446 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da9f181-55c2-40cd-ba0c-61781f00784d-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:10.746183 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.746158 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da9f181-55c2-40cd-ba0c-61781f00784d-kube-api-access-pzbc5" (OuterVolumeSpecName: "kube-api-access-pzbc5") pod "1da9f181-55c2-40cd-ba0c-61781f00784d" (UID: "1da9f181-55c2-40cd-ba0c-61781f00784d"). InnerVolumeSpecName "kube-api-access-pzbc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:29:10.746269 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.746198 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1da9f181-55c2-40cd-ba0c-61781f00784d" (UID: "1da9f181-55c2-40cd-ba0c-61781f00784d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:29:10.814786 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.814756 2556 generic.go:356] "Generic (PLEG): container finished" podID="1da9f181-55c2-40cd-ba0c-61781f00784d" containerID="bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046" exitCode=0 Jul 09 22:29:10.814959 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.814805 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" Jul 09 22:29:10.814959 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.814825 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" event={"ID":"1da9f181-55c2-40cd-ba0c-61781f00784d","Type":"ContainerDied","Data":"bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046"} Jul 09 22:29:10.814959 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.814867 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h" event={"ID":"1da9f181-55c2-40cd-ba0c-61781f00784d","Type":"ContainerDied","Data":"f89c9538c0bedd6869031371bb6a7157ccee641bb4abee7d3a4ff7d79706a041"} Jul 09 22:29:10.814959 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.814888 2556 scope.go:117] "RemoveContainer" containerID="bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046" Jul 09 22:29:10.832759 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.832729 2556 scope.go:117] "RemoveContainer" containerID="bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046" Jul 09 22:29:10.833033 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:29:10.833015 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046\": container with ID starting with bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046 not found: ID does not exist" containerID="bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046" Jul 09 22:29:10.833097 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.833039 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046"} err="failed to get container status \"bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046\": rpc error: code = NotFound desc = could not find container \"bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046\": container with ID starting with bc6e5a185d2737d0e106e5efd603b87e098965378228819f28c47143af923046 not found: ID does not exist" Jul 09 22:29:10.844380 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.844353 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h"] Jul 09 22:29:10.845218 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.845200 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da9f181-55c2-40cd-ba0c-61781f00784d-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:10.845274 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.845223 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzbc5\" (UniqueName: \"kubernetes.io/projected/1da9f181-55c2-40cd-ba0c-61781f00784d-kube-api-access-pzbc5\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:10.848299 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:10.848277 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-03e1d1-8f7457ccd-6wk5h"] Jul 09 22:29:11.276188 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:11.276155 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da9f181-55c2-40cd-ba0c-61781f00784d" path="/var/lib/kubelet/pods/1da9f181-55c2-40cd-ba0c-61781f00784d/volumes" Jul 09 22:29:20.357950 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.357912 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5"] Jul 09 22:29:20.358517 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.358501 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da9f181-55c2-40cd-ba0c-61781f00784d" containerName="maas-api" Jul 09 22:29:20.358517 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.358518 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da9f181-55c2-40cd-ba0c-61781f00784d" containerName="maas-api" Jul 09 22:29:20.358687 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.358544 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae866552-4a1d-4987-b68d-93668d647e11" containerName="maas-api" Jul 09 22:29:20.358687 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.358552 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae866552-4a1d-4987-b68d-93668d647e11" containerName="maas-api" Jul 09 22:29:20.358687 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.358657 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da9f181-55c2-40cd-ba0c-61781f00784d" containerName="maas-api" Jul 09 22:29:20.358687 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.358671 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae866552-4a1d-4987-b68d-93668d647e11" containerName="maas-api" Jul 09 22:29:20.363493 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.363468 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:20.368590 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.368556 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-eefed0-serving-cert\"" Jul 09 22:29:20.381511 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.381484 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5"] Jul 09 22:29:20.531447 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.531415 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-openshift-service-ca\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:20.531645 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.531456 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:20.531645 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.531553 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vs7\" (UniqueName: \"kubernetes.io/projected/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-kube-api-access-27vs7\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:20.632232 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.632148 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-openshift-service-ca\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:20.632232 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.632186 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:20.632232 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.632231 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vs7\" (UniqueName: \"kubernetes.io/projected/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-kube-api-access-27vs7\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:20.632445 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:29:20.632368 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-serving-cert: secret "maas-api-e2e-api-a-eefed0-serving-cert" not found Jul 09 22:29:20.632482 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:29:20.632455 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls podName:1f03eb05-4c4f-4ac1-acf0-c5f47d53394e nodeName:}" failed. No retries permitted until 2026-07-09 22:29:21.132433569 +0000 UTC m=+2280.443051086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls") pod "maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" (UID: "1f03eb05-4c4f-4ac1-acf0-c5f47d53394e") : secret "maas-api-e2e-api-a-eefed0-serving-cert" not found Jul 09 22:29:20.632950 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.632932 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-openshift-service-ca\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:20.643158 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:20.643131 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vs7\" (UniqueName: \"kubernetes.io/projected/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-kube-api-access-27vs7\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:21.137872 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:21.137836 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:21.140224 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:21.140199 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls\") pod \"maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:21.273522 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:21.273488 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:21.413410 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:21.413382 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5"] Jul 09 22:29:21.415177 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:29:21.415152 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f03eb05_4c4f_4ac1_acf0_c5f47d53394e.slice/crio-8a36c2aad69cbb767c1e3c79644cbad164bf5ee73d16e1fb7e5926b58bcf3e27 WatchSource:0}: Error finding container 8a36c2aad69cbb767c1e3c79644cbad164bf5ee73d16e1fb7e5926b58bcf3e27: Status 404 returned error can't find the container with id 8a36c2aad69cbb767c1e3c79644cbad164bf5ee73d16e1fb7e5926b58bcf3e27 Jul 09 22:29:21.880038 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:21.880004 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" event={"ID":"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e","Type":"ContainerStarted","Data":"8a36c2aad69cbb767c1e3c79644cbad164bf5ee73d16e1fb7e5926b58bcf3e27"} Jul 09 22:29:22.886899 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:22.886854 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" event={"ID":"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e","Type":"ContainerStarted","Data":"93a42cd011e7cb8ec1484252a3e00717a47d8da917e6c48baaf1461f0fc53ced"} Jul 09 22:29:22.887279 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:22.886996 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:22.905785 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:22.905739 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" podStartSLOduration=2.373870347 podStartE2EDuration="2.905725018s" totalImagesPullingTime="531.854671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:29:22.904117163 +0000 UTC m=+2282.214734689" watchObservedRunningTime="2026-07-09 22:29:22.905725018 +0000 UTC m=+2282.216342540" Jul 09 22:29:28.416254 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.416218 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p"] Jul 09 22:29:28.420941 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.420914 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.423943 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.423916 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c4e995-serving-cert\"" Jul 09 22:29:28.435426 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.435404 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p"] Jul 09 22:29:28.509188 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.509155 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.509377 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.509271 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1213148b-3f8c-4ad4-a2cb-879667edf5dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.509377 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.509345 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxww\" (UniqueName: \"kubernetes.io/projected/1213148b-3f8c-4ad4-a2cb-879667edf5dc-kube-api-access-skxww\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.610617 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.610557 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1213148b-3f8c-4ad4-a2cb-879667edf5dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.610781 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.610666 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxww\" (UniqueName: \"kubernetes.io/projected/1213148b-3f8c-4ad4-a2cb-879667edf5dc-kube-api-access-skxww\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.610781 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.610698 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.610908 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:29:28.610891 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-serving-cert: secret "maas-api-e2e-api-b-c4e995-serving-cert" not found Jul 09 22:29:28.610961 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:29:28.610951 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls podName:1213148b-3f8c-4ad4-a2cb-879667edf5dc nodeName:}" failed. No retries permitted until 2026-07-09 22:29:29.110937412 +0000 UTC m=+2288.421554912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls") pod "maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" (UID: "1213148b-3f8c-4ad4-a2cb-879667edf5dc") : secret "maas-api-e2e-api-b-c4e995-serving-cert" not found Jul 09 22:29:28.611228 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.611210 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1213148b-3f8c-4ad4-a2cb-879667edf5dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.622085 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.622061 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxww\" (UniqueName: \"kubernetes.io/projected/1213148b-3f8c-4ad4-a2cb-879667edf5dc-kube-api-access-skxww\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:28.898882 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:28.898853 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:29.115323 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:29.115288 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:29.117728 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:29.117701 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls\") pod \"maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:29.335707 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:29.335667 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:29.494672 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:29.494638 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p"] Jul 09 22:29:29.495447 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:29:29.495420 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1213148b_3f8c_4ad4_a2cb_879667edf5dc.slice/crio-47bb5a586add23279205506bcb4b6b325e49043246bb510c257fe9dd9538b36b WatchSource:0}: Error finding container 47bb5a586add23279205506bcb4b6b325e49043246bb510c257fe9dd9538b36b: Status 404 returned error can't find the container with id 47bb5a586add23279205506bcb4b6b325e49043246bb510c257fe9dd9538b36b Jul 09 22:29:29.925520 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:29.925492 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" event={"ID":"1213148b-3f8c-4ad4-a2cb-879667edf5dc","Type":"ContainerStarted","Data":"47bb5a586add23279205506bcb4b6b325e49043246bb510c257fe9dd9538b36b"} Jul 09 22:29:30.932651 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:30.932610 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" event={"ID":"1213148b-3f8c-4ad4-a2cb-879667edf5dc","Type":"ContainerStarted","Data":"633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8"} Jul 09 22:29:30.933046 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:30.932702 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:30.953411 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:30.953363 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" podStartSLOduration=2.5761740680000003 podStartE2EDuration="2.953348505s" totalImagesPullingTime="377.174437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:29:30.95086665 +0000 UTC m=+2290.261484201" watchObservedRunningTime="2026-07-09 22:29:30.953348505 +0000 UTC m=+2290.263966067" Jul 09 22:29:36.944172 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:36.944144 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:41.839149 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:41.839120 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5"] Jul 09 22:29:41.839785 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:41.839390 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" podUID="1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" containerName="maas-api" containerID="cri-o://93a42cd011e7cb8ec1484252a3e00717a47d8da917e6c48baaf1461f0fc53ced" gracePeriod=30 Jul 09 22:29:41.996244 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:41.996206 2556 generic.go:356] "Generic (PLEG): container finished" podID="1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" containerID="93a42cd011e7cb8ec1484252a3e00717a47d8da917e6c48baaf1461f0fc53ced" exitCode=0 Jul 09 22:29:41.996415 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:41.996272 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" event={"ID":"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e","Type":"ContainerDied","Data":"93a42cd011e7cb8ec1484252a3e00717a47d8da917e6c48baaf1461f0fc53ced"} Jul 09 22:29:42.102885 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.102822 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:42.247886 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.247851 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls\") pod \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " Jul 09 22:29:42.248071 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.247904 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-openshift-service-ca\") pod \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " Jul 09 22:29:42.248071 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.247938 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27vs7\" (UniqueName: \"kubernetes.io/projected/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-kube-api-access-27vs7\") pod \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\" (UID: \"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e\") " Jul 09 22:29:42.248261 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.248238 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" (UID: "1f03eb05-4c4f-4ac1-acf0-c5f47d53394e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:29:42.249955 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.249933 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" (UID: "1f03eb05-4c4f-4ac1-acf0-c5f47d53394e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:29:42.250047 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.249995 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-kube-api-access-27vs7" (OuterVolumeSpecName: "kube-api-access-27vs7") pod "1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" (UID: "1f03eb05-4c4f-4ac1-acf0-c5f47d53394e"). InnerVolumeSpecName "kube-api-access-27vs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:29:42.349199 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.349160 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:42.349358 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.349305 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:42.349358 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:42.349330 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vs7\" (UniqueName: \"kubernetes.io/projected/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e-kube-api-access-27vs7\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:43.002775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:43.002746 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" Jul 09 22:29:43.002775 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:43.002762 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5" event={"ID":"1f03eb05-4c4f-4ac1-acf0-c5f47d53394e","Type":"ContainerDied","Data":"8a36c2aad69cbb767c1e3c79644cbad164bf5ee73d16e1fb7e5926b58bcf3e27"} Jul 09 22:29:43.003251 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:43.002816 2556 scope.go:117] "RemoveContainer" containerID="93a42cd011e7cb8ec1484252a3e00717a47d8da917e6c48baaf1461f0fc53ced" Jul 09 22:29:43.025944 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:43.025917 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5"] Jul 09 22:29:43.030342 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:43.030314 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-eefed0-74c4d7f54f-56gj5"] Jul 09 22:29:43.276285 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:43.276200 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" path="/var/lib/kubelet/pods/1f03eb05-4c4f-4ac1-acf0-c5f47d53394e/volumes" Jul 09 22:29:49.137392 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.137359 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p"] Jul 09 22:29:49.137836 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.137600 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" podUID="1213148b-3f8c-4ad4-a2cb-879667edf5dc" containerName="maas-api" containerID="cri-o://633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8" gracePeriod=30 Jul 09 22:29:49.398104 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.398046 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:49.519854 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.519822 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skxww\" (UniqueName: \"kubernetes.io/projected/1213148b-3f8c-4ad4-a2cb-879667edf5dc-kube-api-access-skxww\") pod \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " Jul 09 22:29:49.520036 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.519860 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls\") pod \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " Jul 09 22:29:49.520036 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.519884 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1213148b-3f8c-4ad4-a2cb-879667edf5dc-openshift-service-ca\") pod \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\" (UID: \"1213148b-3f8c-4ad4-a2cb-879667edf5dc\") " Jul 09 22:29:49.520324 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.520292 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1213148b-3f8c-4ad4-a2cb-879667edf5dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1213148b-3f8c-4ad4-a2cb-879667edf5dc" (UID: "1213148b-3f8c-4ad4-a2cb-879667edf5dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:29:49.521972 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.521950 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1213148b-3f8c-4ad4-a2cb-879667edf5dc" (UID: "1213148b-3f8c-4ad4-a2cb-879667edf5dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:29:49.522070 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.522019 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1213148b-3f8c-4ad4-a2cb-879667edf5dc-kube-api-access-skxww" (OuterVolumeSpecName: "kube-api-access-skxww") pod "1213148b-3f8c-4ad4-a2cb-879667edf5dc" (UID: "1213148b-3f8c-4ad4-a2cb-879667edf5dc"). InnerVolumeSpecName "kube-api-access-skxww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:29:49.621365 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.621336 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxww\" (UniqueName: \"kubernetes.io/projected/1213148b-3f8c-4ad4-a2cb-879667edf5dc-kube-api-access-skxww\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:49.621365 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.621362 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1213148b-3f8c-4ad4-a2cb-879667edf5dc-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:49.621365 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:49.621371 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1213148b-3f8c-4ad4-a2cb-879667edf5dc-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:29:50.042999 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.042964 2556 generic.go:356] "Generic (PLEG): container finished" podID="1213148b-3f8c-4ad4-a2cb-879667edf5dc" containerID="633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8" exitCode=0 Jul 09 22:29:50.043167 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.043010 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" Jul 09 22:29:50.043167 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.043048 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" event={"ID":"1213148b-3f8c-4ad4-a2cb-879667edf5dc","Type":"ContainerDied","Data":"633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8"} Jul 09 22:29:50.043167 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.043099 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p" event={"ID":"1213148b-3f8c-4ad4-a2cb-879667edf5dc","Type":"ContainerDied","Data":"47bb5a586add23279205506bcb4b6b325e49043246bb510c257fe9dd9538b36b"} Jul 09 22:29:50.043167 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.043123 2556 scope.go:117] "RemoveContainer" containerID="633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8" Jul 09 22:29:50.052361 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.052349 2556 scope.go:117] "RemoveContainer" containerID="633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8" Jul 09 22:29:50.052644 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:29:50.052621 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8\": container with ID starting with 633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8 not found: ID does not exist" containerID="633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8" Jul 09 22:29:50.052753 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.052649 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8"} err="failed to get container status \"633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8\": rpc error: code = NotFound desc = could not find container \"633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8\": container with ID starting with 633ff7dcf1b4ba9952c448e0c6a827f2617d5bee56b48e94d1765d9beff589e8 not found: ID does not exist" Jul 09 22:29:50.065446 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.065424 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p"] Jul 09 22:29:50.069806 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:50.069785 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c4e995-fd6d5c596-7j95p"] Jul 09 22:29:51.277145 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:51.277115 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1213148b-3f8c-4ad4-a2cb-879667edf5dc" path="/var/lib/kubelet/pods/1213148b-3f8c-4ad4-a2cb-879667edf5dc/volumes" Jul 09 22:29:59.604595 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.601989 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj"] Jul 09 22:29:59.604595 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.603040 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1213148b-3f8c-4ad4-a2cb-879667edf5dc" containerName="maas-api" Jul 09 22:29:59.604595 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.603058 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1213148b-3f8c-4ad4-a2cb-879667edf5dc" containerName="maas-api" Jul 09 22:29:59.604595 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.603112 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" containerName="maas-api" Jul 09 22:29:59.604595 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.603123 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" containerName="maas-api" Jul 09 22:29:59.604595 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.603479 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f03eb05-4c4f-4ac1-acf0-c5f47d53394e" containerName="maas-api" Jul 09 22:29:59.604595 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.603503 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1213148b-3f8c-4ad4-a2cb-879667edf5dc" containerName="maas-api" Jul 09 22:29:59.613092 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.613069 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:29:59.615934 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.615886 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e4b894-serving-cert\"" Jul 09 22:29:59.618931 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.618903 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f70bd9-d457-47a0-9d06-ab1e60416f01-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:29:59.619109 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.618965 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:29:59.619109 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.619012 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5wzj\" (UniqueName: \"kubernetes.io/projected/53f70bd9-d457-47a0-9d06-ab1e60416f01-kube-api-access-m5wzj\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:29:59.619974 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.619949 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj"] Jul 09 22:29:59.720230 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.720199 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f70bd9-d457-47a0-9d06-ab1e60416f01-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:29:59.720436 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.720241 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:29:59.720436 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.720260 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5wzj\" (UniqueName: \"kubernetes.io/projected/53f70bd9-d457-47a0-9d06-ab1e60416f01-kube-api-access-m5wzj\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:29:59.720436 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:29:59.720352 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-serving-cert: secret "maas-api-e2e-shared-a-e4b894-serving-cert" not found Jul 09 22:29:59.720436 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:29:59.720423 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls podName:53f70bd9-d457-47a0-9d06-ab1e60416f01 nodeName:}" failed. No retries permitted until 2026-07-09 22:30:00.220406322 +0000 UTC m=+2319.531023827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls") pod "maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" (UID: "53f70bd9-d457-47a0-9d06-ab1e60416f01") : secret "maas-api-e2e-shared-a-e4b894-serving-cert" not found Jul 09 22:29:59.720902 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.720881 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f70bd9-d457-47a0-9d06-ab1e60416f01-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:29:59.728848 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:29:59.728826 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5wzj\" (UniqueName: \"kubernetes.io/projected/53f70bd9-d457-47a0-9d06-ab1e60416f01-kube-api-access-m5wzj\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:30:00.225146 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:00.225097 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:30:00.227598 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:00.227535 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls\") pod \"maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:30:00.525380 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:00.525289 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:30:00.690812 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:00.690781 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj"] Jul 09 22:30:00.692251 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:30:00.692220 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f70bd9_d457_47a0_9d06_ab1e60416f01.slice/crio-75b738f1ec7213e568842d9a84d9b985ff2b7aa4a16c671a6bf10de43d8dfe52 WatchSource:0}: Error finding container 75b738f1ec7213e568842d9a84d9b985ff2b7aa4a16c671a6bf10de43d8dfe52: Status 404 returned error can't find the container with id 75b738f1ec7213e568842d9a84d9b985ff2b7aa4a16c671a6bf10de43d8dfe52 Jul 09 22:30:01.106648 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:01.106611 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" event={"ID":"53f70bd9-d457-47a0-9d06-ab1e60416f01","Type":"ContainerStarted","Data":"75b738f1ec7213e568842d9a84d9b985ff2b7aa4a16c671a6bf10de43d8dfe52"} Jul 09 22:30:02.114046 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:02.114010 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" event={"ID":"53f70bd9-d457-47a0-9d06-ab1e60416f01","Type":"ContainerStarted","Data":"85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db"} Jul 09 22:30:02.114469 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:02.114129 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:30:02.134558 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:02.134506 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" podStartSLOduration=2.638308824 podStartE2EDuration="3.134490614s" totalImagesPullingTime="496.18179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:30:02.132009692 +0000 UTC m=+2321.442627217" watchObservedRunningTime="2026-07-09 22:30:02.134490614 +0000 UTC m=+2321.445108137" Jul 09 22:30:07.650051 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.650011 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x"] Jul 09 22:30:07.654288 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.654262 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:07.657025 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.656988 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ecf4af-serving-cert\"" Jul 09 22:30:07.667314 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.667286 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x"] Jul 09 22:30:07.690004 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.689974 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znl9s\" (UniqueName: \"kubernetes.io/projected/c0900929-bb7f-402e-8b85-1e9479116fd8-kube-api-access-znl9s\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:07.690202 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.690024 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:07.690202 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.690097 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0900929-bb7f-402e-8b85-1e9479116fd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:07.790919 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.790880 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:07.791131 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.790982 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0900929-bb7f-402e-8b85-1e9479116fd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:07.791131 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:30:07.791046 2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-serving-cert: secret "maas-api-e2e-shared-b-ecf4af-serving-cert" not found Jul 09 22:30:07.791131 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.791081 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znl9s\" (UniqueName: \"kubernetes.io/projected/c0900929-bb7f-402e-8b85-1e9479116fd8-kube-api-access-znl9s\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:07.791131 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:30:07.791104 2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls podName:c0900929-bb7f-402e-8b85-1e9479116fd8 nodeName:}" failed. No retries permitted until 2026-07-09 22:30:08.291088658 +0000 UTC m=+2327.601706165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls") pod "maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" (UID: "c0900929-bb7f-402e-8b85-1e9479116fd8") : secret "maas-api-e2e-shared-b-ecf4af-serving-cert" not found Jul 09 22:30:07.791665 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.791644 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0900929-bb7f-402e-8b85-1e9479116fd8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:07.807048 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:07.807021 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znl9s\" (UniqueName: \"kubernetes.io/projected/c0900929-bb7f-402e-8b85-1e9479116fd8-kube-api-access-znl9s\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:08.126463 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:08.126429 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:30:08.297761 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:08.297716 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:08.300179 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:08.300142 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls\") pod \"maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:08.566448 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:08.566363 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:08.717766 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:08.717548 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x"] Jul 09 22:30:08.719703 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:30:08.719678 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0900929_bb7f_402e_8b85_1e9479116fd8.slice/crio-e7cc2cb774b776d1139c69ab025c8c4f42562a2841f934a488160d6a92196a73 WatchSource:0}: Error finding container e7cc2cb774b776d1139c69ab025c8c4f42562a2841f934a488160d6a92196a73: Status 404 returned error can't find the container with id e7cc2cb774b776d1139c69ab025c8c4f42562a2841f934a488160d6a92196a73 Jul 09 22:30:09.156694 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:09.156660 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" event={"ID":"c0900929-bb7f-402e-8b85-1e9479116fd8","Type":"ContainerStarted","Data":"383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0"} Jul 09 22:30:09.156856 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:09.156716 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" event={"ID":"c0900929-bb7f-402e-8b85-1e9479116fd8","Type":"ContainerStarted","Data":"e7cc2cb774b776d1139c69ab025c8c4f42562a2841f934a488160d6a92196a73"} Jul 09 22:30:09.156856 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:09.156771 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:30:09.187484 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:09.187348 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" podStartSLOduration=1.830015961 podStartE2EDuration="2.187329039s" totalImagesPullingTime="357.313078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:30:09.186604287 +0000 UTC m=+2328.497221801" watchObservedRunningTime="2026-07-09 22:30:09.187329039 +0000 UTC m=+2328.497946563" Jul 09 22:30:15.168202 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:30:15.168128 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:31:21.381447 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:31:21.381345 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:31:21.385622 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:31:21.385602 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:33:18.134097 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.134019 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj"] Jul 09 22:33:18.134644 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.134255 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" podUID="53f70bd9-d457-47a0-9d06-ab1e60416f01" containerName="maas-api" containerID="cri-o://85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db" gracePeriod=30 Jul 09 22:33:18.382005 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.381981 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:33:18.495232 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.495142 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f70bd9-d457-47a0-9d06-ab1e60416f01-openshift-service-ca\") pod \"53f70bd9-d457-47a0-9d06-ab1e60416f01\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " Jul 09 22:33:18.495377 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.495233 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5wzj\" (UniqueName: \"kubernetes.io/projected/53f70bd9-d457-47a0-9d06-ab1e60416f01-kube-api-access-m5wzj\") pod \"53f70bd9-d457-47a0-9d06-ab1e60416f01\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " Jul 09 22:33:18.495377 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.495282 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls\") pod \"53f70bd9-d457-47a0-9d06-ab1e60416f01\" (UID: \"53f70bd9-d457-47a0-9d06-ab1e60416f01\") " Jul 09 22:33:18.495556 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.495532 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f70bd9-d457-47a0-9d06-ab1e60416f01-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53f70bd9-d457-47a0-9d06-ab1e60416f01" (UID: "53f70bd9-d457-47a0-9d06-ab1e60416f01"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:33:18.497218 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.497192 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f70bd9-d457-47a0-9d06-ab1e60416f01-kube-api-access-m5wzj" (OuterVolumeSpecName: "kube-api-access-m5wzj") pod "53f70bd9-d457-47a0-9d06-ab1e60416f01" (UID: "53f70bd9-d457-47a0-9d06-ab1e60416f01"). InnerVolumeSpecName "kube-api-access-m5wzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:33:18.497290 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.497265 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53f70bd9-d457-47a0-9d06-ab1e60416f01" (UID: "53f70bd9-d457-47a0-9d06-ab1e60416f01"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:33:18.596223 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.596189 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53f70bd9-d457-47a0-9d06-ab1e60416f01-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:33:18.596223 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.596216 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53f70bd9-d457-47a0-9d06-ab1e60416f01-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:33:18.596223 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:18.596229 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5wzj\" (UniqueName: \"kubernetes.io/projected/53f70bd9-d457-47a0-9d06-ab1e60416f01-kube-api-access-m5wzj\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:33:19.217846 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.217812 2556 generic.go:356] "Generic (PLEG): container finished" podID="53f70bd9-d457-47a0-9d06-ab1e60416f01" containerID="85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db" exitCode=0 Jul 09 22:33:19.218271 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.217860 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" Jul 09 22:33:19.218271 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.217895 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" event={"ID":"53f70bd9-d457-47a0-9d06-ab1e60416f01","Type":"ContainerDied","Data":"85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db"} Jul 09 22:33:19.218271 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.217949 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj" event={"ID":"53f70bd9-d457-47a0-9d06-ab1e60416f01","Type":"ContainerDied","Data":"75b738f1ec7213e568842d9a84d9b985ff2b7aa4a16c671a6bf10de43d8dfe52"} Jul 09 22:33:19.218271 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.217969 2556 scope.go:117] "RemoveContainer" containerID="85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db" Jul 09 22:33:19.227694 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.227672 2556 scope.go:117] "RemoveContainer" containerID="85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db" Jul 09 22:33:19.227976 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:33:19.227955 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db\": container with ID starting with 85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db not found: ID does not exist" containerID="85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db" Jul 09 22:33:19.228042 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.227983 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db"} err="failed to get container status \"85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db\": rpc error: code = NotFound desc = could not find container \"85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db\": container with ID starting with 85ffd6dc083d40b11fe2656e270557e04145c08d829f950426090c6dac20b0db not found: ID does not exist" Jul 09 22:33:19.252126 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.252097 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj"] Jul 09 22:33:19.264378 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.264356 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e4b894-7d8fd6f9fc-zmkkj"] Jul 09 22:33:19.275563 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:19.275530 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f70bd9-d457-47a0-9d06-ab1e60416f01" path="/var/lib/kubelet/pods/53f70bd9-d457-47a0-9d06-ab1e60416f01/volumes" Jul 09 22:33:24.653283 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.653249 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x"] Jul 09 22:33:24.653677 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.653491 2556 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" podUID="c0900929-bb7f-402e-8b85-1e9479116fd8" containerName="maas-api" containerID="cri-o://383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0" gracePeriod=30 Jul 09 22:33:24.904872 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.904812 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:33:24.964440 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.964407 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0900929-bb7f-402e-8b85-1e9479116fd8-openshift-service-ca\") pod \"c0900929-bb7f-402e-8b85-1e9479116fd8\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " Jul 09 22:33:24.964622 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.964487 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znl9s\" (UniqueName: \"kubernetes.io/projected/c0900929-bb7f-402e-8b85-1e9479116fd8-kube-api-access-znl9s\") pod \"c0900929-bb7f-402e-8b85-1e9479116fd8\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " Jul 09 22:33:24.964695 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.964655 2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls\") pod \"c0900929-bb7f-402e-8b85-1e9479116fd8\" (UID: \"c0900929-bb7f-402e-8b85-1e9479116fd8\") " Jul 09 22:33:24.964788 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.964768 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0900929-bb7f-402e-8b85-1e9479116fd8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0900929-bb7f-402e-8b85-1e9479116fd8" (UID: "c0900929-bb7f-402e-8b85-1e9479116fd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:33:24.964987 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.964965 2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0900929-bb7f-402e-8b85-1e9479116fd8-openshift-service-ca\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:33:24.966479 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.966447 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0900929-bb7f-402e-8b85-1e9479116fd8-kube-api-access-znl9s" (OuterVolumeSpecName: "kube-api-access-znl9s") pod "c0900929-bb7f-402e-8b85-1e9479116fd8" (UID: "c0900929-bb7f-402e-8b85-1e9479116fd8"). InnerVolumeSpecName "kube-api-access-znl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:33:24.966591 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:24.966525 2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0900929-bb7f-402e-8b85-1e9479116fd8" (UID: "c0900929-bb7f-402e-8b85-1e9479116fd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:33:25.066065 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.066034 2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znl9s\" (UniqueName: \"kubernetes.io/projected/c0900929-bb7f-402e-8b85-1e9479116fd8-kube-api-access-znl9s\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:33:25.066065 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.066060 2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0900929-bb7f-402e-8b85-1e9479116fd8-maas-api-tls\") on node \"ip-10-0-143-160.ec2.internal\" DevicePath \"\"" Jul 09 22:33:25.253006 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.252920 2556 generic.go:356] "Generic (PLEG): container finished" podID="c0900929-bb7f-402e-8b85-1e9479116fd8" containerID="383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0" exitCode=0 Jul 09 22:33:25.253006 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.252963 2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" Jul 09 22:33:25.253219 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.253000 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" event={"ID":"c0900929-bb7f-402e-8b85-1e9479116fd8","Type":"ContainerDied","Data":"383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0"} Jul 09 22:33:25.253219 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.253052 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x" event={"ID":"c0900929-bb7f-402e-8b85-1e9479116fd8","Type":"ContainerDied","Data":"e7cc2cb774b776d1139c69ab025c8c4f42562a2841f934a488160d6a92196a73"} Jul 09 22:33:25.253219 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.253074 2556 scope.go:117] "RemoveContainer" containerID="383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0" Jul 09 22:33:25.262508 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.262489 2556 scope.go:117] "RemoveContainer" containerID="383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0" Jul 09 22:33:25.262740 ip-10-0-143-160 kubenswrapper[2556]: E0709 22:33:25.262723 2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0\": container with ID starting with 383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0 not found: ID does not exist" containerID="383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0" Jul 09 22:33:25.262803 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.262749 2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0"} err="failed to get container status \"383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0\": rpc error: code = NotFound desc = could not find container \"383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0\": container with ID starting with 383129bb030c3711facd7d4b86eb5be0e7ff77f78b26077a64671c1af5474fb0 not found: ID does not exist" Jul 09 22:33:25.276077 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.276045 2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x"] Jul 09 22:33:25.282903 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:25.282882 2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ecf4af-6f8d4c8465-q7b4x"] Jul 09 22:33:27.280653 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:27.280542 2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0900929-bb7f-402e-8b85-1e9479116fd8" path="/var/lib/kubelet/pods/c0900929-bb7f-402e-8b85-1e9479116fd8/volumes" Jul 09 22:33:35.992530 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:35.992496 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-768b466dcc-vwgvf_ad82804c-00c4-4dc9-bcb3-2060628bebbc/manager/0.log" Jul 09 22:33:36.344343 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:36.344318 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85dfbddc47-dsbqf_de54257a-58b6-4c6b-989c-4bc460a32922/copy-manifests/0.log" Jul 09 22:33:36.351372 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:36.351353 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85dfbddc47-dsbqf_de54257a-58b6-4c6b-989c-4bc460a32922/manager/0.log" Jul 09 22:33:37.381319 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.381293 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv_3330a7ab-1f3c-46b8-8ed3-91671393a529/extract/0.log" Jul 09 22:33:37.388036 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.388014 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv_3330a7ab-1f3c-46b8-8ed3-91671393a529/util/0.log" Jul 09 22:33:37.395424 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.395406 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv_3330a7ab-1f3c-46b8-8ed3-91671393a529/pull/0.log" Jul 09 22:33:37.509848 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.509816 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9_50dc060c-16ec-4b9d-9d6b-7ca254ef792c/util/0.log" Jul 09 22:33:37.517554 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.517524 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9_50dc060c-16ec-4b9d-9d6b-7ca254ef792c/pull/0.log" Jul 09 22:33:37.524029 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.524013 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9_50dc060c-16ec-4b9d-9d6b-7ca254ef792c/extract/0.log" Jul 09 22:33:37.643700 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.643619 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq_8f23b965-be1e-4bfe-bcb2-e7c68237fee8/pull/0.log" Jul 09 22:33:37.650723 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.650699 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq_8f23b965-be1e-4bfe-bcb2-e7c68237fee8/extract/0.log" Jul 09 22:33:37.657948 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.657927 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq_8f23b965-be1e-4bfe-bcb2-e7c68237fee8/util/0.log" Jul 09 22:33:37.775151 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.775121 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7_3271ffea-ab07-40e3-a0b8-5a73e24f920f/extract/0.log" Jul 09 22:33:37.781782 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.781751 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7_3271ffea-ab07-40e3-a0b8-5a73e24f920f/util/0.log" Jul 09 22:33:37.788729 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:37.788704 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7_3271ffea-ab07-40e3-a0b8-5a73e24f920f/pull/0.log" Jul 09 22:33:39.108781 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:39.108753 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46nkvx_14fc4f3d-f43e-4d5c-839c-902301586b07/istio-proxy/0.log" Jul 09 22:33:39.227915 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:39.227885 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w9qxn_68ddcb3f-a771-48a3-bc49-75c332ea3532/discovery/0.log" Jul 09 22:33:39.797599 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:39.797539 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-zg8bh_89d5f6cf-36e8-4565-ae62-d2e7e28c9af0/payload-processing/0.log" Jul 09 22:33:39.908978 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:39.908948 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b5486fbd7-n8ft6_139e71b1-5134-4bc2-8a29-96f5dc16c5b6/router/0.log" Jul 09 22:33:40.367222 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:40.367197 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-fhcqz_6e5ef9ab-78d0-4fa3-afbc-c107879f6c1c/main/0.log" Jul 09 22:33:40.595867 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:40.595837 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbt6vps_c1e57e32-00d3-4f03-83b0-9c151743559b/main/0.log" Jul 09 22:33:48.432385 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:48.432353 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bjk4m_a2e19c4a-e0b3-46a7-b8de-563c6b248018/global-pull-secret-syncer/0.log" Jul 09 22:33:48.616979 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:48.616948 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fvpk9_1c4578ac-8173-4dbc-8332-250b5a40fd7a/konnectivity-agent/0.log" Jul 09 22:33:48.730044 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:48.729962 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-160.ec2.internal_487f50e765b2ad11dd8e605acdc1e631/haproxy/0.log" Jul 09 22:33:52.835172 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:52.835140 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv_3330a7ab-1f3c-46b8-8ed3-91671393a529/extract/0.log" Jul 09 22:33:52.860513 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:52.860488 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv_3330a7ab-1f3c-46b8-8ed3-91671393a529/util/0.log" Jul 09 22:33:52.884687 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:52.884654 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zshqv_3330a7ab-1f3c-46b8-8ed3-91671393a529/pull/0.log" Jul 09 22:33:52.919309 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:52.919282 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9_50dc060c-16ec-4b9d-9d6b-7ca254ef792c/extract/0.log" Jul 09 22:33:52.946253 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:52.946222 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9_50dc060c-16ec-4b9d-9d6b-7ca254ef792c/util/0.log" Jul 09 22:33:52.970853 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:52.970832 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rwjl9_50dc060c-16ec-4b9d-9d6b-7ca254ef792c/pull/0.log" Jul 09 22:33:53.002235 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:53.002213 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq_8f23b965-be1e-4bfe-bcb2-e7c68237fee8/extract/0.log" Jul 09 22:33:53.027773 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:53.027752 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq_8f23b965-be1e-4bfe-bcb2-e7c68237fee8/util/0.log" Jul 09 22:33:53.069257 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:53.069234 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73h8qrq_8f23b965-be1e-4bfe-bcb2-e7c68237fee8/pull/0.log" Jul 09 22:33:53.108055 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:53.107996 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7_3271ffea-ab07-40e3-a0b8-5a73e24f920f/extract/0.log" Jul 09 22:33:53.134165 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:53.134143 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7_3271ffea-ab07-40e3-a0b8-5a73e24f920f/util/0.log" Jul 09 22:33:53.167338 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:53.167310 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mhhb7_3271ffea-ab07-40e3-a0b8-5a73e24f920f/pull/0.log" Jul 09 22:33:55.468464 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.468431 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_031b392d-83f4-493c-a7c4-1eb8af8e2b19/alertmanager/0.log" Jul 09 22:33:55.492354 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.492326 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_031b392d-83f4-493c-a7c4-1eb8af8e2b19/config-reloader/0.log" Jul 09 22:33:55.515182 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.515156 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_031b392d-83f4-493c-a7c4-1eb8af8e2b19/kube-rbac-proxy-web/0.log" Jul 09 22:33:55.559419 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.559393 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_031b392d-83f4-493c-a7c4-1eb8af8e2b19/kube-rbac-proxy/0.log" Jul 09 22:33:55.587460 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.587437 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_031b392d-83f4-493c-a7c4-1eb8af8e2b19/kube-rbac-proxy-metric/0.log" Jul 09 22:33:55.611784 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.611760 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_031b392d-83f4-493c-a7c4-1eb8af8e2b19/prom-label-proxy/0.log" Jul 09 22:33:55.633746 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.633722 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_031b392d-83f4-493c-a7c4-1eb8af8e2b19/init-config-reloader/0.log" Jul 09 22:33:55.895035 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.895009 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7f248_c48e0e65-afa9-4b13-9992-c575253f6260/node-exporter/0.log" Jul 09 22:33:55.927463 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.927440 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7f248_c48e0e65-afa9-4b13-9992-c575253f6260/kube-rbac-proxy/0.log" Jul 09 22:33:55.971791 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:55.971772 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7f248_c48e0e65-afa9-4b13-9992-c575253f6260/init-textfile/0.log" Jul 09 22:33:56.177430 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.177373 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-98pn6_f51bbc95-3cb3-4f86-b1b2-de3022e7126d/kube-rbac-proxy-main/0.log" Jul 09 22:33:56.200092 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.200070 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-98pn6_f51bbc95-3cb3-4f86-b1b2-de3022e7126d/kube-rbac-proxy-self/0.log" Jul 09 22:33:56.223611 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.223567 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-98pn6_f51bbc95-3cb3-4f86-b1b2-de3022e7126d/openshift-state-metrics/0.log" Jul 09 22:33:56.452472 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.452392 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-jp4f9_117fe50e-09df-4baa-99f5-cbb6f561d195/prometheus-operator/0.log" Jul 09 22:33:56.480325 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.480293 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-jp4f9_117fe50e-09df-4baa-99f5-cbb6f561d195/kube-rbac-proxy/0.log" Jul 09 22:33:56.546759 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.546715 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bcb67bb6c-dtdst_a48308d8-19a9-485a-9116-8726093d7174/telemeter-client/0.log" Jul 09 22:33:56.573468 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.573447 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bcb67bb6c-dtdst_a48308d8-19a9-485a-9116-8726093d7174/reload/0.log" Jul 09 22:33:56.598548 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.598527 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bcb67bb6c-dtdst_a48308d8-19a9-485a-9116-8726093d7174/kube-rbac-proxy/0.log" Jul 09 22:33:56.638438 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.638412 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5457574644-9b7q9_5f9b054f-21a3-404c-967a-1fd09a092cda/thanos-query/0.log" Jul 09 22:33:56.660700 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.660674 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5457574644-9b7q9_5f9b054f-21a3-404c-967a-1fd09a092cda/kube-rbac-proxy-web/0.log" Jul 09 22:33:56.682686 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.682668 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5457574644-9b7q9_5f9b054f-21a3-404c-967a-1fd09a092cda/kube-rbac-proxy/0.log" Jul 09 22:33:56.705920 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.705854 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5457574644-9b7q9_5f9b054f-21a3-404c-967a-1fd09a092cda/prom-label-proxy/0.log" Jul 09 22:33:56.730002 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.729979 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5457574644-9b7q9_5f9b054f-21a3-404c-967a-1fd09a092cda/kube-rbac-proxy-rules/0.log" Jul 09 22:33:56.753272 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:56.753251 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5457574644-9b7q9_5f9b054f-21a3-404c-967a-1fd09a092cda/kube-rbac-proxy-metrics/0.log" Jul 09 22:33:58.563694 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.563658 2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb"] Jul 09 22:33:58.564243 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.564223 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f70bd9-d457-47a0-9d06-ab1e60416f01" containerName="maas-api" Jul 09 22:33:58.564334 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.564246 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f70bd9-d457-47a0-9d06-ab1e60416f01" containerName="maas-api" Jul 09 22:33:58.564334 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.564262 2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0900929-bb7f-402e-8b85-1e9479116fd8" containerName="maas-api" Jul 09 22:33:58.564334 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.564270 2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0900929-bb7f-402e-8b85-1e9479116fd8" containerName="maas-api" Jul 09 22:33:58.564487 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.564377 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f70bd9-d457-47a0-9d06-ab1e60416f01" containerName="maas-api" Jul 09 22:33:58.564487 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.564393 2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0900929-bb7f-402e-8b85-1e9479116fd8" containerName="maas-api" Jul 09 22:33:58.567460 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.567438 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.570137 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.570118 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rgg8m\"/\"openshift-service-ca.crt\"" Jul 09 22:33:58.571076 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.571058 2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rgg8m\"/\"default-dockercfg-ctdb7\"" Jul 09 22:33:58.571164 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.571080 2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rgg8m\"/\"kube-root-ca.crt\"" Jul 09 22:33:58.577145 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.577127 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb"] Jul 09 22:33:58.672520 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.672491 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-sys\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.672707 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.672528 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-lib-modules\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.672707 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.672550 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-podres\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.672707 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.672641 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-proc\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.672707 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.672702 2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4b7q\" (UniqueName: \"kubernetes.io/projected/db379436-dc2b-42a6-a26b-1ce8eb5b947d-kube-api-access-v4b7q\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.742037 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.742010 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/1.log" Jul 09 22:33:58.746642 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.746624 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xfqh4_4f715282-1ab7-427e-973a-550762cade1e/console-operator/2.log" Jul 09 22:33:58.773760 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.773739 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-sys\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.773832 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.773768 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-lib-modules\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.773832 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.773784 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-podres\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.773910 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.773855 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-sys\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.773956 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.773942 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-podres\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.773991 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.773953 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-lib-modules\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.773991 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.773969 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-proc\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.774057 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.773997 2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4b7q\" (UniqueName: \"kubernetes.io/projected/db379436-dc2b-42a6-a26b-1ce8eb5b947d-kube-api-access-v4b7q\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.774091 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.774056 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/db379436-dc2b-42a6-a26b-1ce8eb5b947d-proc\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.783961 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.783934 2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4b7q\" (UniqueName: \"kubernetes.io/projected/db379436-dc2b-42a6-a26b-1ce8eb5b947d-kube-api-access-v4b7q\") pod \"perf-node-gather-daemonset-89rkb\" (UID: \"db379436-dc2b-42a6-a26b-1ce8eb5b947d\") " pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:58.878400 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:58.878362 2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:59.013904 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:59.013823 2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb"] Jul 09 22:33:59.016081 ip-10-0-143-160 kubenswrapper[2556]: W0709 22:33:59.016048 2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddb379436_dc2b_42a6_a26b_1ce8eb5b947d.slice/crio-5e1ad103e79bd03d1f3bfe9bc5df39c932793186c0ddff2fd96a3c0654ed2cc3 WatchSource:0}: Error finding container 5e1ad103e79bd03d1f3bfe9bc5df39c932793186c0ddff2fd96a3c0654ed2cc3: Status 404 returned error can't find the container with id 5e1ad103e79bd03d1f3bfe9bc5df39c932793186c0ddff2fd96a3c0654ed2cc3 Jul 09 22:33:59.349523 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:59.349496 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-86lx5_25c8a8d3-26e2-484c-ba48-0b79b989ee38/download-server/0.log" Jul 09 22:33:59.442362 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:59.442320 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" event={"ID":"db379436-dc2b-42a6-a26b-1ce8eb5b947d","Type":"ContainerStarted","Data":"62e0f888f719c8e00ee5918e72809fa19e5194126e02ee604c911f348dc9d609"} Jul 09 22:33:59.442513 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:59.442487 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:33:59.442598 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:59.442536 2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" event={"ID":"db379436-dc2b-42a6-a26b-1ce8eb5b947d","Type":"ContainerStarted","Data":"5e1ad103e79bd03d1f3bfe9bc5df39c932793186c0ddff2fd96a3c0654ed2cc3"} Jul 09 22:33:59.470370 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:33:59.470320 2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" podStartSLOduration=1.470304892 podStartE2EDuration="1.470304892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:33:59.469862302 +0000 UTC m=+2558.780479824" watchObservedRunningTime="2026-07-09 22:33:59.470304892 +0000 UTC m=+2558.780922415" Jul 09 22:34:00.025720 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:00.025690 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-cl9rc_3760cd42-b37d-490a-85ef-4c446d344a5e/volume-data-source-validator/0.log" Jul 09 22:34:01.251493 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:01.251468 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4sg8_4488e1dc-e74a-48a7-a208-e6371f9d4375/dns/0.log" Jul 09 22:34:01.274698 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:01.274675 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b4sg8_4488e1dc-e74a-48a7-a208-e6371f9d4375/kube-rbac-proxy/0.log" Jul 09 22:34:01.374041 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:01.374006 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9znf9_a2ca5280-d3ac-4cec-82ed-67dcdfff85f8/dns-node-resolver/0.log" Jul 09 22:34:01.933872 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:01.933835 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-678cb6686-wvkh5_30f64e04-7199-470a-86ca-cd96a6fb3021/registry/0.log" Jul 09 22:34:01.963863 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:01.963835 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dp58g_95277ca8-ebc1-4f6e-ae65-c9daf9b5d16b/node-ca/0.log" Jul 09 22:34:02.982183 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:02.982152 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46nkvx_14fc4f3d-f43e-4d5c-839c-902301586b07/istio-proxy/0.log" Jul 09 22:34:03.083643 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:03.083612 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w9qxn_68ddcb3f-a771-48a3-bc49-75c332ea3532/discovery/0.log" Jul 09 22:34:03.310759 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:03.310659 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-zg8bh_89d5f6cf-36e8-4565-ae62-d2e7e28c9af0/payload-processing/0.log" Jul 09 22:34:03.332448 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:03.332391 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b5486fbd7-n8ft6_139e71b1-5134-4bc2-8a29-96f5dc16c5b6/router/0.log" Jul 09 22:34:04.005998 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:04.005953 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8n2cw_115dcab9-3d51-43fa-ab57-c00a61331fc3/serve-healthcheck-canary/0.log" Jul 09 22:34:04.550562 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:04.550532 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-8q4rh_04d3e78f-c77a-4677-8b22-643f3e657964/insights-operator/0.log" Jul 09 22:34:04.551004 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:04.550990 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-8q4rh_04d3e78f-c77a-4677-8b22-643f3e657964/insights-operator/1.log" Jul 09 22:34:04.575000 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:04.574976 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2677l_d5894dba-53ef-4271-838c-3420e112755b/kube-rbac-proxy/0.log" Jul 09 22:34:04.601352 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:04.601323 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2677l_d5894dba-53ef-4271-838c-3420e112755b/exporter/0.log" Jul 09 22:34:04.623127 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:04.623095 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2677l_d5894dba-53ef-4271-838c-3420e112755b/extractor/0.log" Jul 09 22:34:05.457677 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:05.457652 2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rgg8m/perf-node-gather-daemonset-89rkb" Jul 09 22:34:07.083028 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:07.083000 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-768b466dcc-vwgvf_ad82804c-00c4-4dc9-bcb3-2060628bebbc/manager/0.log" Jul 09 22:34:07.224697 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:07.224672 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85dfbddc47-dsbqf_de54257a-58b6-4c6b-989c-4bc460a32922/manager/0.log" Jul 09 22:34:07.251318 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:07.251294 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85dfbddc47-dsbqf_de54257a-58b6-4c6b-989c-4bc460a32922/copy-manifests/0.log" Jul 09 22:34:08.715803 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:08.715767 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7b9cdbd59d-f5c8f_eeb51403-af15-4663-9fb6-d86d137c196e/manager/0.log" Jul 09 22:34:09.083186 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:09.083160 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46nkvx_14fc4f3d-f43e-4d5c-839c-902301586b07/istio-proxy/0.log" Jul 09 22:34:09.176548 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:09.176519 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w9qxn_68ddcb3f-a771-48a3-bc49-75c332ea3532/discovery/0.log" Jul 09 22:34:09.356676 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:09.356648 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-zg8bh_89d5f6cf-36e8-4565-ae62-d2e7e28c9af0/payload-processing/0.log" Jul 09 22:34:09.371454 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:09.371433 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b5486fbd7-n8ft6_139e71b1-5134-4bc2-8a29-96f5dc16c5b6/router/0.log" Jul 09 22:34:10.574003 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:10.573974 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf46nkvx_14fc4f3d-f43e-4d5c-839c-902301586b07/istio-proxy/0.log" Jul 09 22:34:10.658736 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:10.658705 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w9qxn_68ddcb3f-a771-48a3-bc49-75c332ea3532/discovery/0.log" Jul 09 22:34:10.835953 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:10.835925 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-zg8bh_89d5f6cf-36e8-4565-ae62-d2e7e28c9af0/payload-processing/0.log" Jul 09 22:34:10.855694 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:10.855669 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b5486fbd7-n8ft6_139e71b1-5134-4bc2-8a29-96f5dc16c5b6/router/0.log" Jul 09 22:34:14.474547 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:14.474513 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-6kzqk_520a37a8-e07d-41c7-99c5-51bcbdd6ef23/kube-storage-version-migrator-operator/1.log" Jul 09 22:34:14.476058 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:14.476031 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-6kzqk_520a37a8-e07d-41c7-99c5-51bcbdd6ef23/kube-storage-version-migrator-operator/0.log" Jul 09 22:34:15.871077 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:15.871050 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hzrbj_82a91008-923a-4f71-9420-b743b9a391e9/kube-multus-additional-cni-plugins/0.log" Jul 09 22:34:15.894858 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:15.894833 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hzrbj_82a91008-923a-4f71-9420-b743b9a391e9/egress-router-binary-copy/0.log" Jul 09 22:34:15.917094 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:15.917072 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hzrbj_82a91008-923a-4f71-9420-b743b9a391e9/cni-plugins/0.log" Jul 09 22:34:15.939137 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:15.939110 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hzrbj_82a91008-923a-4f71-9420-b743b9a391e9/bond-cni-plugin/0.log" Jul 09 22:34:15.960169 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:15.960135 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hzrbj_82a91008-923a-4f71-9420-b743b9a391e9/routeoverride-cni/0.log" Jul 09 22:34:15.981029 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:15.981003 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hzrbj_82a91008-923a-4f71-9420-b743b9a391e9/whereabouts-cni-bincopy/0.log" Jul 09 22:34:16.002878 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:16.002856 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hzrbj_82a91008-923a-4f71-9420-b743b9a391e9/whereabouts-cni/0.log" Jul 09 22:34:16.038765 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:16.038737 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bntnp_fd78623e-cb49-4f50-ae98-3eb77e2c762e/kube-multus/0.log" Jul 09 22:34:16.136581 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:16.136549 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5jcvb_ceeea23b-5ed9-45c1-97c0-c223a39f03b9/network-metrics-daemon/0.log" Jul 09 22:34:16.160250 ip-10-0-143-160 kubenswrapper[2556]: I0709 22:34:16.160227 2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5jcvb_ceeea23b-5ed9-45c1-97c0-c223a39f03b9/kube-rbac-proxy/0.log"