Jul 14 19:45:58.220124 ip-10-0-133-243 systemd[1]: Starting Kubernetes Kubelet... Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.673889 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679808 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679820 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679823 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679839 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679842 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679845 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679847 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:45:58.688552 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679850 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679853 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679855 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679858 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679861 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679863 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679866 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679869 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679871 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679874 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679876 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679879 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679882 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679885 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679889 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679891 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679894 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679897 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679899 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679901 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:45:58.703125 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679904 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679907 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679912 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679915 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679918 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679922 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679927 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679930 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679933 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679936 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679938 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679941 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679945 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679949 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679952 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679954 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679957 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679960 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679962 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:45:58.704916 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679965 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679967 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679970 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679972 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679975 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679978 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679981 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679984 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679986 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679989 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679991 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679994 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679996 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.679998 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680001 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680004 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680007 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680010 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680014 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680017 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:45:58.707259 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680020 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680023 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680026 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680028 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680031 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680034 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680036 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680038 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680042 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680045 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680047 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680050 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680052 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680055 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680058 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680061 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680064 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680067 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680069 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680072 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:45:58.718761 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680074 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680077 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680079 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680082 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680085 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680087 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680090 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680092 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680095 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680098 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680104 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680107 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680110 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680775 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680781 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680784 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680787 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680789 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680792 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680795 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:45:58.721122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680797 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680800 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680802 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680805 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680807 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680810 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680813 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680816 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680818 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680821 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680823 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680843 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680845 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680848 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680850 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680853 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680855 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680858 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680861 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680864 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:45:58.722973 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680866 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680869 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680874 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680877 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680880 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680882 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680885 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680887 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680890 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680893 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680895 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680898 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680901 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680905 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680909 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680912 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680914 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680917 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680920 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:45:58.725257 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680923 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680926 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680929 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680931 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680934 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680936 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680939 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680942 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680944 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680947 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680949 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680952 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680954 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680957 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680959 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680963 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680966 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680969 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680971 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680974 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:45:58.734756 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680976 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680978 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680981 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680984 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680987 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680989 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680991 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680994 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680996 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.680999 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681001 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681003 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681006 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681009 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681011 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681014 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681016 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681018 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681021 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:45:58.736920 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681023 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681028 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681031 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681033 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681036 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681038 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681040 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681044 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681048 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681051 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681053 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681056 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681061 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681064 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681433 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681438 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681441 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681443 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681446 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:45:58.738639 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681449 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681451 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681455 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681459 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681462 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681464 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681467 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681470 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681473 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681487 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681491 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681494 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681496 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681499 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681501 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681504 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681506 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681509 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681511 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:45:58.751363 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681533 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:45:58.750661 ip-10-0-133-243 systemd[1]: Started Kubernetes Kubelet. Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681536 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681540 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681543 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681545 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681548 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681551 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681554 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681556 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681559 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681561 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681564 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681566 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681569 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681571 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681574 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681577 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681579 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681582 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681585 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:45:59.000166 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681587 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681600 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681603 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681607 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681610 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681613 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681615 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681618 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681621 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681624 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681627 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681629 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681632 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681636 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681641 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681643 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681646 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681648 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681651 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681654 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:45:59.036621 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681656 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681659 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681661 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681664 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681666 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681669 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681672 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681674 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681677 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681679 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681682 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681684 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681688 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681690 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681693 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681696 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681699 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681701 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681703 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:45:59.043053 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681706 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681708 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681711 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681713 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681716 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681718 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681722 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681726 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681729 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681733 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681736 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681738 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681741 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681743 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681746 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.681748 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681839 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681854 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681860 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681864 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 19:45:59.043710 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681868 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681871 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681876 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681880 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681883 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681886 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681889 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681893 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681896 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681899 2578 flags.go:64] FLAG: --cgroup-root="" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681902 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681905 2578 flags.go:64] FLAG: --client-ca-file="" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681908 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.681911 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682726 2578 flags.go:64] FLAG: --cluster-domain="" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682730 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682734 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682737 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682742 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682747 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682750 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682753 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682757 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682760 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 14 19:45:59.044300 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682762 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682765 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682768 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682771 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682775 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682778 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682781 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682784 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682787 2578 flags.go:64] FLAG: --enable-server="true" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682789 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682794 2578 flags.go:64] FLAG: --event-burst="100" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682797 2578 flags.go:64] FLAG: --event-qps="50" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682800 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682803 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682806 2578 flags.go:64] FLAG: --eviction-hard="" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682809 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682812 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682815 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682819 2578 flags.go:64] FLAG: --eviction-soft="" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682822 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682837 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682840 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682843 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682846 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682849 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 19:45:59.045022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682852 2578 flags.go:64] FLAG: --feature-gates="" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682857 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682861 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682864 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682867 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682870 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682873 2578 flags.go:64] FLAG: --help="false" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682886 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682889 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682892 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682896 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682899 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682903 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682905 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682908 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682911 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682914 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682916 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682919 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682922 2578 flags.go:64] FLAG: --kube-reserved="" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682925 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682928 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682931 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682933 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 19:45:59.045792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682936 2578 flags.go:64] FLAG: --lock-file="" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682945 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682948 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682951 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682956 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682959 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682961 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682964 2578 flags.go:64] FLAG: --logging-format="text" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682967 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682971 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682981 2578 flags.go:64] FLAG: --manifest-url="" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682984 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682988 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682991 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682994 2578 flags.go:64] FLAG: --max-pods="110" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.682997 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683000 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683003 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683006 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683008 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683011 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683014 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683020 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683023 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683026 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 19:45:59.046547 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683029 2578 flags.go:64] FLAG: --pod-cidr="" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683032 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683037 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683040 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683043 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683046 2578 flags.go:64] FLAG: --port="10250" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683048 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683051 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-042a82d5756228adc" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683054 2578 flags.go:64] FLAG: --qos-reserved="" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683063 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683066 2578 flags.go:64] FLAG: --register-node="true" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683069 2578 flags.go:64] FLAG: --register-with-taints="" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683072 2578 flags.go:64] FLAG: --registry-burst="10" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683075 2578 flags.go:64] FLAG: --registry-qps="5" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683077 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683080 2578 flags.go:64] FLAG: --reserved-memory="" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683085 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683088 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683092 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683095 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683098 2578 flags.go:64] FLAG: --runonce="false" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683101 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683104 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683106 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683109 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 19:45:59.047260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683112 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683115 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683118 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683121 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683123 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683126 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683129 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683131 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683134 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683137 2578 flags.go:64] FLAG: --system-cgroups="" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683140 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683145 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683147 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683150 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683159 2578 flags.go:64] FLAG: --tls-min-version="" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683162 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683165 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683168 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683171 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683174 2578 flags.go:64] FLAG: --v="2" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683177 2578 flags.go:64] FLAG: --version="false" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683181 2578 flags.go:64] FLAG: --vmodule="" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683184 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683189 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 19:45:59.048004 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683300 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683305 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683308 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683311 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683314 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683316 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683319 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683322 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683324 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683327 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683329 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683332 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683334 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683337 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683339 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683342 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683345 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683347 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683350 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683352 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:45:59.048671 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683354 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683357 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683359 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683362 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683364 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683366 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683370 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683372 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683376 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683379 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683382 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683386 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683388 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683392 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683395 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683397 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683399 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683402 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683404 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:45:59.049300 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683407 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683409 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683411 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683414 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683416 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683418 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683421 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683423 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683426 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683428 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683430 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683433 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683435 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683438 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683441 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683443 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683446 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683448 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683451 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:45:59.049931 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683455 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683458 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683461 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683464 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683467 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683471 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683473 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683477 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683480 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683482 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683485 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683487 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683490 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683492 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683495 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683497 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683500 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683502 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683504 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683507 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:45:59.050544 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683509 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683512 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683514 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683517 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683519 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683522 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683524 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683526 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683529 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683532 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683536 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683538 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683541 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683544 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683547 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683549 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683552 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683556 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683558 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:45:59.051135 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683563 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.683565 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.683573 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.690938 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.690954 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.690972 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.690979 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691036 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691041 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691045 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691048 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691051 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691054 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691057 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:45:59.051660 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691060 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691062 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691065 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691068 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691070 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691073 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691077 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691081 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691084 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691087 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691089 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691092 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691094 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691097 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691100 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691102 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691104 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691107 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691110 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691112 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:45:59.052137 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691115 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691118 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691122 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691125 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691128 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691131 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691135 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691138 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691141 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691144 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691147 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691150 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691154 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691156 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691159 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691162 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691164 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691167 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691170 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691172 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:45:59.052691 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691175 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691178 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691180 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691183 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691186 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691188 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691190 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691193 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691195 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691198 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691200 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691203 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691206 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691209 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691211 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691218 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691221 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691224 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691227 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691230 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:45:59.053254 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691234 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691237 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691239 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691242 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691245 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691247 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691250 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691252 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691255 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691257 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691260 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691262 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691265 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691268 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691270 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691272 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691275 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691277 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691280 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:45:59.053988 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691282 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691286 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691289 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691292 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691295 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691298 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691301 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691303 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691306 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691309 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691311 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691315 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691318 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.691324 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:45:59.054609 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691430 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691435 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691438 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691441 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691444 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691446 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691449 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691452 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691456 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691460 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691463 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691466 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691468 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691471 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691474 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691476 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691479 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691481 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691483 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 19:45:59.055240 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691486 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691488 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691491 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691493 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691496 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691498 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691501 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691503 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691505 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691508 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691512 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691514 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691516 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691520 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691523 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691525 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691528 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691530 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691533 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691535 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691537 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 19:45:59.254536 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691540 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691542 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691545 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691547 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691550 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691552 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691555 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691557 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691560 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691562 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691564 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691567 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691569 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691572 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691574 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691577 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691579 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691581 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691584 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691586 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 19:45:59.255602 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691589 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691593 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691595 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691598 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691602 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691605 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691608 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691611 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691613 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691616 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691618 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691621 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691623 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691626 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691628 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691631 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691633 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691635 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691638 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 19:45:59.258495 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691640 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691643 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691645 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691647 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691650 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691652 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691655 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691657 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691659 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691662 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691664 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691667 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691669 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691672 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691676 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691679 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691681 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691685 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691688 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 19:45:59.259062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:45:58.691690 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.691695 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.691796 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.694050 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.695621 2578 server.go:1019] "Starting client certificate rotation" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.695718 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.695754 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.698496 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.702744 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.715594 2578 log.go:25] "Validated CRI v1 runtime API" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.718878 2578 log.go:25] "Validated CRI v1 image API" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.719686 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 19:45:59.259565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.723236 2578 fs.go:135] Filesystem UUIDs: map[35a17f1c-b225-4ae8-b01b-3f36910d44f1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d456369c-8777-4b9d-9472-9f565b384b11:/dev/nvme0n1p3] Jul 14 19:45:59.259954 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.723250 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 19:45:59.259954 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.728928 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.729149 2578 manager.go:216] Machine: {Timestamp:2026-07-14 19:45:58.727794691 +0000 UTC m=+0.375358246 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3166494 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c10780ffc1c91f2a79b9c4730f6de SystemUUID:ec2c1078-0ffc-1c91-f2a7-9b9c4730f6de BootID:91ea174e-5c0d-4453-9a6f-5b243d07e51e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3d:c0:43:3d:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3d:c0:43:3d:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:b6:a5:f4:04:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.729247 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.729321 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.732949 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.732983 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-243.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.733766 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.733780 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.733790 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.734637 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.735904 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.739519 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.739618 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.739645 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.739655 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.739672 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.739682 2578 kubelet.go:398] "Adding apiserver pod source" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.739693 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 19:45:59.260115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.740009 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.740025 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.742296 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744058 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744085 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744924 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744936 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744942 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744947 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744956 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744961 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744968 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744975 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744981 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.744986 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.745000 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.745005 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.746936 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.746944 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.749887 2578 server.go:1268] "Started kubelet" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.750062 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.750111 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.750139 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.751456 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.755752 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.755765 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.755762 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-243.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.756532 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.759089 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.760706 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.760711 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.760797 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761350 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761410 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761424 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761441 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761456 2578 factory.go:55] Registering systemd factory Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761464 2578 factory.go:223] Registration of the systemd container factory successfully Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761557 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761565 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761690 2578 factory.go:153] Registering CRI-O factory Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761703 2578 factory.go:223] Registration of the crio container factory successfully Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761723 2578 factory.go:103] Registering Raw factory Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.761736 2578 manager.go:1196] Started watching for new ooms in manager Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.762129 2578 manager.go:349] Starting recovery of all containers Jul 14 19:45:59.260645 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.762273 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-243.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.761960 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-243.ec2.internal.18c23f9ffbda014e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-243.ec2.internal,UID:ip-10-0-133-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-243.ec2.internal,},FirstTimestamp:2026-07-14 19:45:58.749856078 +0000 UTC m=+0.397419638,LastTimestamp:2026-07-14 19:45:58.749856078 +0000 UTC m=+0.397419638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-243.ec2.internal,}" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.764593 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.769627 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.772265 2578 manager.go:354] Recovery completed Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.776672 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.778463 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.778501 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.778511 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.778983 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.778990 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.778992 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.779059 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.779531 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.779555 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.779570 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.780073 2578 policy_none.go:49] "None policy: Start" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.780084 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.780095 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.780522 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-243.ec2.internal.18c23f9ffd8edd8b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-243.ec2.internal,UID:ip-10-0-133-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-243.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-243.ec2.internal,},FirstTimestamp:2026-07-14 19:45:58.778486155 +0000 UTC m=+0.426049712,LastTimestamp:2026-07-14 19:45:58.778486155 +0000 UTC m=+0.426049712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-243.ec2.internal,}" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.781025 2578 policy_none.go:47] "Start" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.789743 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-243.ec2.internal.18c23f9ffd8f2bb6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-243.ec2.internal,UID:ip-10-0-133-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-243.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-243.ec2.internal,},FirstTimestamp:2026-07-14 19:45:58.778506166 +0000 UTC m=+0.426069725,LastTimestamp:2026-07-14 19:45:58.778506166 +0000 UTC m=+0.426069725,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-243.ec2.internal,}" Jul 14 19:45:59.261943 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.798787 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-27jkk" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.798920 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-243.ec2.internal.18c23f9ffd8f4eb5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-243.ec2.internal,UID:ip-10-0-133-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-243.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-243.ec2.internal,},FirstTimestamp:2026-07-14 19:45:58.778515125 +0000 UTC m=+0.426078685,LastTimestamp:2026-07-14 19:45:58.778515125 +0000 UTC m=+0.426078685,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-243.ec2.internal,}" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.805914 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-27jkk" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.811090 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-133-243.ec2.internal.18c23f9ffd8edd8b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-243.ec2.internal.18c23f9ffd8edd8b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-243.ec2.internal,UID:ip-10-0-133-243.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-243.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-243.ec2.internal,},FirstTimestamp:2026-07-14 19:45:58.778486155 +0000 UTC m=+0.426049712,LastTimestamp:2026-07-14 19:45:58.779541104 +0000 UTC m=+0.427104660,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-243.ec2.internal,}" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.861483 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.864449 2578 manager.go:337] "Starting Device Plugin manager" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.864481 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.864490 2578 server.go:87] "Starting device plugin registration server" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.864761 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.864772 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.864884 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.864967 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.864975 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.865488 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.865556 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.876996 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.877035 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.877055 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.877090 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.879164 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.965288 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.966296 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.966324 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.966333 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.966355 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.969460 2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-243.ec2.internal\" not found" node="ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.977373 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.977391 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-243.ec2.internal\": node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.977458 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal"] Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.977512 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.978283 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:45:59.262565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.978307 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.978324 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.979596 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.979695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.979718 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.980274 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.980279 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.980319 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.980294 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.980334 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.980349 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.981417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.981439 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.982175 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientMemory" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.982208 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:58.982221 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeHasSufficientPID" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:58.995218 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.019295 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-243.ec2.internal\" not found" node="ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.022795 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-243.ec2.internal\" not found" node="ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.063598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9010125a2cabcaaf81321ade9e08441-config\") pod \"kube-apiserver-proxy-ip-10-0-133-243.ec2.internal\" (UID: \"f9010125a2cabcaaf81321ade9e08441\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.063744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b72e59c0d82e73db7c035d21a4e6cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal\" (UID: \"2b72e59c0d82e73db7c035d21a4e6cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.063762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b72e59c0d82e73db7c035d21a4e6cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal\" (UID: \"2b72e59c0d82e73db7c035d21a4e6cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.096107 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.164316 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9010125a2cabcaaf81321ade9e08441-config\") pod \"kube-apiserver-proxy-ip-10-0-133-243.ec2.internal\" (UID: \"f9010125a2cabcaaf81321ade9e08441\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.164345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b72e59c0d82e73db7c035d21a4e6cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal\" (UID: \"2b72e59c0d82e73db7c035d21a4e6cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.164363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b72e59c0d82e73db7c035d21a4e6cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal\" (UID: \"2b72e59c0d82e73db7c035d21a4e6cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.263485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.164403 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b72e59c0d82e73db7c035d21a4e6cad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal\" (UID: \"2b72e59c0d82e73db7c035d21a4e6cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.264189 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.164420 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f9010125a2cabcaaf81321ade9e08441-config\") pod \"kube-apiserver-proxy-ip-10-0-133-243.ec2.internal\" (UID: \"f9010125a2cabcaaf81321ade9e08441\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.264189 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.164430 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b72e59c0d82e73db7c035d21a4e6cad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal\" (UID: \"2b72e59c0d82e73db7c035d21a4e6cad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.264189 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.196442 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.435331 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.296719 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.435331 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.321921 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.435331 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.324899 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.435331 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.397088 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.650325 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.497788 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.650325 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.598459 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-243.ec2.internal\" not found" Jul 14 19:45:59.650325 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.632038 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.660877 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.671812 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.673633 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.688284 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.694927 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.695029 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.695070 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.739920 2578 apiserver.go:52] "Watching apiserver" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.760621 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.760802 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.761104 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal","openshift-multus/network-metrics-daemon-mktvj","openshift-network-diagnostics/network-check-target-f4clt","openshift-network-operator/iptables-alerter-rvqps","kube-system/global-pull-secret-syncer-qzk58","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks","openshift-image-registry/node-ca-w2p7r","openshift-multus/multus-6pcwn","openshift-multus/multus-additional-cni-plugins-gb6f2","openshift-ovn-kubernetes/ovnkube-node-kkjx6","kube-system/konnectivity-agent-65rs9","kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9dnkx","openshift-dns/node-resolver-xthjn"] Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.763235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.764893 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xthjn" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.765915 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4xc9b\"" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.766013 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.766152 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.766892 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.767999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-var-lib-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.768036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c688k\" (UniqueName: \"kubernetes.io/projected/fb74204e-abe8-45da-8bbd-97eb68965a34-kube-api-access-c688k\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.768184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-systemd\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.768212 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-log-socket\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.768236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.851916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.768392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.768421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-env-overrides\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.768449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-run-netns\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.769509 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.769955 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.770322 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.771944 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-etc-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.771984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-ovn\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-cni-bin\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-cni-netd\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-ovnkube-config\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-kubelet\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-systemd-units\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772246 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-node-log\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb74204e-abe8-45da-8bbd-97eb68965a34-ovn-node-metrics-cert\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772639 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-ovnkube-script-lib\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772884 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wsd65\"" Jul 14 19:45:59.852975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772943 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-slash\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.772975 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.773236 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7dnwd\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.773337 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.773350 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.773421 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.775076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.776821 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qjvdn\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.777022 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.777037 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.777101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.777685 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.777752 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.778027 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x49s5\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.778075 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.779137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.780557 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.781171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6pcwn" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.783243 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.785674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.785718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.789308 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789368 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g4tpb\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789421 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789475 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789490 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.789534 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789551 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789591 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789604 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789607 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789671 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 19:45:59.854247 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789687 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789878 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789914 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5nfkz\"" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.789977 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tzmln\"" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.790535 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4c6r7\"" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.791541 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.792578 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.794070 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.809817 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 19:40:59 +0000 UTC" deadline="2028-04-23 22:52:16.112435184 +0000 UTC" Jul 14 19:45:59.855367 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.809857 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15579h6m16.302581749s" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.862820 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-ovnkube-config\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-host\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-etc-selinux\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-kubelet\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-etc-kubernetes\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/060c531f-952a-432e-a4a3-c08cd392bdcd-kubelet-config\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-688jg\" (UniqueName: \"kubernetes.io/projected/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-kube-api-access-688jg\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-tuned\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d525fe8-c0cb-44b9-b406-478e58125c9a-tmp\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ssk\" (UniqueName: \"kubernetes.io/projected/9aaae312-ec5a-4cb3-8f65-700e7e980d45-kube-api-access-p8ssk\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c688k\" (UniqueName: \"kubernetes.io/projected/fb74204e-abe8-45da-8bbd-97eb68965a34-kube-api-access-c688k\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873560 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-run\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-device-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-daemon-config\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.020451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-env-overrides\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7fp\" (UniqueName: \"kubernetes.io/projected/9d525fe8-c0cb-44b9-b406-478e58125c9a-kube-api-access-dq7fp\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-sys-fs\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-hostroot\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873755 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-ovnkube-config\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ce272c-d523-4253-bf80-92b16152b6e5-tmp-dir\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-run-netns\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52800f2-1350-49d4-ad60-df31b5d00640-host\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873935 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-run-netns\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.873978 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-system-cni-dir\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-os-release\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874066 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-env-overrides\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874096 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-kubelet\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.021465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb74204e-abe8-45da-8bbd-97eb68965a34-ovn-node-metrics-cert\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-modprobe-d\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-systemd\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874190 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-kubelet\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-registration-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-conf-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874243 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52cpt\" (UniqueName: \"kubernetes.io/projected/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-kube-api-access-52cpt\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/060c531f-952a-432e-a4a3-c08cd392bdcd-dbus\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-var-lib-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txf7w\" (UniqueName: \"kubernetes.io/projected/95f78bfa-f2ce-48a5-a34b-16c4db079895-kube-api-access-txf7w\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-os-release\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874347 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-var-lib-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d17644b4-19e1-4aa9-b8d3-10e6062c5ffd-konnectivity-ca\") pod \"konnectivity-agent-65rs9\" (UID: \"d17644b4-19e1-4aa9-b8d3-10e6062c5ffd\") " pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-iptables-alerter-script\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cnibin\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/18ce272c-d523-4253-bf80-92b16152b6e5-hosts-file\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.022248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffng\" (UniqueName: \"kubernetes.io/projected/18ce272c-d523-4253-bf80-92b16152b6e5-kube-api-access-5ffng\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-log-socket\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874508 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-socket-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874537 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-log-socket\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874557 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-netns\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-etc-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-ovn\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874674 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-etc-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-cni-netd\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-sys\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-ovn\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-socket-dir-parent\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-host-slash\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-systemd-units\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-cni-multus\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.023126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-slash\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874798 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-systemd-units\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874754 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-cni-netd\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysctl-d\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-cnibin\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aaae312-ec5a-4cb3-8f65-700e7e980d45-cni-binary-copy\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-slash\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-cni-bin\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d17644b4-19e1-4aa9-b8d3-10e6062c5ffd-agent-certs\") pod \"konnectivity-agent-65rs9\" (UID: \"d17644b4-19e1-4aa9-b8d3-10e6062c5ffd\") " pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.874975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysctl-conf\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-lib-modules\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a52800f2-1350-49d4-ad60-df31b5d00640-serviceca\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875072 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445sm\" (UniqueName: \"kubernetes.io/projected/a52800f2-1350-49d4-ad60-df31b5d00640-kube-api-access-445sm\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-cni-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-multus-certs\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-cni-bin\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.023874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-node-log\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875210 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-cni-bin\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-ovnkube-script-lib\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-system-cni-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875281 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-openvswitch\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysconfig\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875306 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-node-log\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-kubernetes\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-k8s-cni-cncf-io\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-systemd\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875432 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-run-systemd\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-run-ovn-kubernetes\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-var-lib-kubelet\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.024587 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875502 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb74204e-abe8-45da-8bbd-97eb68965a34-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6px4s\" (UniqueName: \"kubernetes.io/projected/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-kube-api-access-6px4s\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.875673 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fb74204e-abe8-45da-8bbd-97eb68965a34-ovnkube-script-lib\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.877795 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fb74204e-abe8-45da-8bbd-97eb68965a34-ovn-node-metrics-cert\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.887287 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c688k\" (UniqueName: \"kubernetes.io/projected/fb74204e-abe8-45da-8bbd-97eb68965a34-kube-api-access-c688k\") pod \"ovnkube-node-kkjx6\" (UID: \"fb74204e-abe8-45da-8bbd-97eb68965a34\") " pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.940955 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.943487 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l449g" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.950814 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l449g" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.975917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7fp\" (UniqueName: \"kubernetes.io/projected/9d525fe8-c0cb-44b9-b406-478e58125c9a-kube-api-access-dq7fp\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.975948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-sys-fs\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.975968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-hostroot\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.975984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ce272c-d523-4253-bf80-92b16152b6e5-tmp-dir\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976017 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52800f2-1350-49d4-ad60-df31b5d00640-host\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-system-cni-dir\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-os-release\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-hostroot\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976114 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:00.025361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976137 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-system-cni-dir\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976200 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-os-release\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976238 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52800f2-1350-49d4-ad60-df31b5d00640-host\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-modprobe-d\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-systemd\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-registration-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-conf-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52cpt\" (UniqueName: \"kubernetes.io/projected/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-kube-api-access-52cpt\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/060c531f-952a-432e-a4a3-c08cd392bdcd-dbus\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txf7w\" (UniqueName: \"kubernetes.io/projected/95f78bfa-f2ce-48a5-a34b-16c4db079895-kube-api-access-txf7w\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-conf-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-os-release\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d17644b4-19e1-4aa9-b8d3-10e6062c5ffd-konnectivity-ca\") pod \"konnectivity-agent-65rs9\" (UID: \"d17644b4-19e1-4aa9-b8d3-10e6062c5ffd\") " pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-registration-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cni-binary-copy\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976455 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-sys-fs\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.026083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-iptables-alerter-script\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976569 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-os-release\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976637 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ce272c-d523-4253-bf80-92b16152b6e5-tmp-dir\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cnibin\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976640 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976704 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-modprobe-d\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/18ce272c-d523-4253-bf80-92b16152b6e5-hosts-file\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976745 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/18ce272c-d523-4253-bf80-92b16152b6e5-hosts-file\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffng\" (UniqueName: \"kubernetes.io/projected/18ce272c-d523-4253-bf80-92b16152b6e5-kube-api-access-5ffng\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-systemd\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-socket-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976831 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/060c531f-952a-432e-a4a3-c08cd392bdcd-dbus\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-netns\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976758 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-cnibin\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-sys\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976930 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-socket-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-socket-dir-parent\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.027667 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.976969 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-host-slash\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.976990 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-netns\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-cni-multus\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977050 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-cni-multus\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977053 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-socket-dir-parent\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.977055 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs podName:8ff66e5b-b8e1-43f9-bdba-b82de3668ef3 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:00.477017828 +0000 UTC m=+2.124581373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs") pod "network-metrics-daemon-mktvj" (UID: "8ff66e5b-b8e1-43f9-bdba-b82de3668ef3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977050 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-sys\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977093 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-host-slash\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysctl-d\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977122 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-iptables-alerter-script\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-cnibin\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977155 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-cnibin\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aaae312-ec5a-4cb3-8f65-700e7e980d45-cni-binary-copy\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-cni-bin\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d17644b4-19e1-4aa9-b8d3-10e6062c5ffd-agent-certs\") pod \"konnectivity-agent-65rs9\" (UID: \"d17644b4-19e1-4aa9-b8d3-10e6062c5ffd\") " pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.028349 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977264 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-cni-bin\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysctl-conf\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977265 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysctl-d\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-lib-modules\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977316 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a52800f2-1350-49d4-ad60-df31b5d00640-serviceca\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445sm\" (UniqueName: \"kubernetes.io/projected/a52800f2-1350-49d4-ad60-df31b5d00640-kube-api-access-445sm\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-cni-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977391 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysctl-conf\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-multus-certs\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-lib-modules\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977433 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-system-cni-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977492 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysconfig\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-kubernetes\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.029887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977584 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-system-cni-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977493 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-cni-dir\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-multus-certs\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-k8s-cni-cncf-io\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-sysconfig\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-var-lib-kubelet\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977680 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-run-k8s-cni-cncf-io\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6px4s\" (UniqueName: \"kubernetes.io/projected/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-kube-api-access-6px4s\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-host\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977729 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d17644b4-19e1-4aa9-b8d3-10e6062c5ffd-konnectivity-ca\") pod \"konnectivity-agent-65rs9\" (UID: \"d17644b4-19e1-4aa9-b8d3-10e6062c5ffd\") " pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-etc-selinux\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-kubernetes\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-kubelet\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977788 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9aaae312-ec5a-4cb3-8f65-700e7e980d45-cni-binary-copy\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-var-lib-kubelet\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977813 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-host-var-lib-kubelet\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977838 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-etc-kubernetes\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977851 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-etc-selinux\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.030744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/060c531f-952a-432e-a4a3-c08cd392bdcd-kubelet-config\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977871 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-host\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-688jg\" (UniqueName: \"kubernetes.io/projected/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-kube-api-access-688jg\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977920 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/060c531f-952a-432e-a4a3-c08cd392bdcd-kubelet-config\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9aaae312-ec5a-4cb3-8f65-700e7e980d45-etc-kubernetes\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.977948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-tuned\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d525fe8-c0cb-44b9-b406-478e58125c9a-tmp\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ssk\" (UniqueName: \"kubernetes.io/projected/9aaae312-ec5a-4cb3-8f65-700e7e980d45-kube-api-access-p8ssk\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978182 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-run\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978229 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9d525fe8-c0cb-44b9-b406-478e58125c9a-run\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-device-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.978266 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-daemon-config\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.978333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret podName:060c531f-952a-432e-a4a3-c08cd392bdcd nodeName:}" failed. No retries permitted until 2026-07-14 19:46:00.478313865 +0000 UTC m=+2.125877411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret") pod "global-pull-secret-syncer-qzk58" (UID: "060c531f-952a-432e-a4a3-c08cd392bdcd") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978337 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a52800f2-1350-49d4-ad60-df31b5d00640-serviceca\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.978388 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/95f78bfa-f2ce-48a5-a34b-16c4db079895-device-dir\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.031387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.979176 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9aaae312-ec5a-4cb3-8f65-700e7e980d45-multus-daemon-config\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.979957 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d17644b4-19e1-4aa9-b8d3-10e6062c5ffd-agent-certs\") pod \"konnectivity-agent-65rs9\" (UID: \"d17644b4-19e1-4aa9-b8d3-10e6062c5ffd\") " pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.980136 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9d525fe8-c0cb-44b9-b406-478e58125c9a-etc-tuned\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.980413 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d525fe8-c0cb-44b9-b406-478e58125c9a-tmp\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.989444 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.989463 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.989473 2578 projected.go:196] Error preparing data for projected volume kube-api-access-w2f2m for pod openshift-network-diagnostics/network-check-target-f4clt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:45:59.989523 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m podName:c0cf5e05-c1b6-4453-aea9-7f5629f80f4c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:00.489509435 +0000 UTC m=+2.137072979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w2f2m" (UniqueName: "kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m") pod "network-check-target-f4clt" (UID: "c0cf5e05-c1b6-4453-aea9-7f5629f80f4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.992640 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6px4s\" (UniqueName: \"kubernetes.io/projected/fbce3123-7485-48fa-b9fd-3b3e1e43a9ee-kube-api-access-6px4s\") pod \"multus-additional-cni-plugins-gb6f2\" (UID: \"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee\") " pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.993007 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52cpt\" (UniqueName: \"kubernetes.io/projected/5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43-kube-api-access-52cpt\") pod \"iptables-alerter-rvqps\" (UID: \"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43\") " pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.999685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7fp\" (UniqueName: \"kubernetes.io/projected/9d525fe8-c0cb-44b9-b406-478e58125c9a-kube-api-access-dq7fp\") pod \"tuned-9dnkx\" (UID: \"9d525fe8-c0cb-44b9-b406-478e58125c9a\") " pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.999751 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffng\" (UniqueName: \"kubernetes.io/projected/18ce272c-d523-4253-bf80-92b16152b6e5-kube-api-access-5ffng\") pod \"node-resolver-xthjn\" (UID: \"18ce272c-d523-4253-bf80-92b16152b6e5\") " pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.999800 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-445sm\" (UniqueName: \"kubernetes.io/projected/a52800f2-1350-49d4-ad60-df31b5d00640-kube-api-access-445sm\") pod \"node-ca-w2p7r\" (UID: \"a52800f2-1350-49d4-ad60-df31b5d00640\") " pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.999907 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ssk\" (UniqueName: \"kubernetes.io/projected/9aaae312-ec5a-4cb3-8f65-700e7e980d45-kube-api-access-p8ssk\") pod \"multus-6pcwn\" (UID: \"9aaae312-ec5a-4cb3-8f65-700e7e980d45\") " pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.999929 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txf7w\" (UniqueName: \"kubernetes.io/projected/95f78bfa-f2ce-48a5-a34b-16c4db079895-kube-api-access-txf7w\") pod \"aws-ebs-csi-driver-node-s89ks\" (UID: \"95f78bfa-f2ce-48a5-a34b-16c4db079895\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.032037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:45:59.999963 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-688jg\" (UniqueName: \"kubernetes.io/projected/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-kube-api-access-688jg\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.060017 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9010125a2cabcaaf81321ade9e08441.slice/crio-a43ed94eee4e1359b2885a5dcdfba877de25bee60f51f71a74476f567fe54f3b WatchSource:0}: Error finding container a43ed94eee4e1359b2885a5dcdfba877de25bee60f51f71a74476f567fe54f3b: Status 404 returned error can't find the container with id a43ed94eee4e1359b2885a5dcdfba877de25bee60f51f71a74476f567fe54f3b Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.060870 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b72e59c0d82e73db7c035d21a4e6cad.slice/crio-b6998147c1a4de84a47f082cd7015292c5a3f820957281cc19fd4a48a1d3a94b WatchSource:0}: Error finding container b6998147c1a4de84a47f082cd7015292c5a3f820957281cc19fd4a48a1d3a94b: Status 404 returned error can't find the container with id b6998147c1a4de84a47f082cd7015292c5a3f820957281cc19fd4a48a1d3a94b Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.065037 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.085877 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.097065 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb74204e_abe8_45da_8bbd_97eb68965a34.slice/crio-77ada98ee0dfe7b071c3fa057655a362048753cb590fbd7a3c177a85eb2d03ef WatchSource:0}: Error finding container 77ada98ee0dfe7b071c3fa057655a362048753cb590fbd7a3c177a85eb2d03ef: Status 404 returned error can't find the container with id 77ada98ee0dfe7b071c3fa057655a362048753cb590fbd7a3c177a85eb2d03ef Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.111767 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xthjn" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.122295 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ce272c_d523_4253_bf80_92b16152b6e5.slice/crio-af43d13d486d9378a1d934749e6cf6a8c23623f30e13efdbe975a3c69060bcaa WatchSource:0}: Error finding container af43d13d486d9378a1d934749e6cf6a8c23623f30e13efdbe975a3c69060bcaa: Status 404 returned error can't find the container with id af43d13d486d9378a1d934749e6cf6a8c23623f30e13efdbe975a3c69060bcaa Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.138407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.141103 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.147544 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d525fe8_c0cb_44b9_b406_478e58125c9a.slice/crio-0320639a3276a6eae13ee467fe91337f701315631e39e4aa03067e100f820235 WatchSource:0}: Error finding container 0320639a3276a6eae13ee467fe91337f701315631e39e4aa03067e100f820235: Status 404 returned error can't find the container with id 0320639a3276a6eae13ee467fe91337f701315631e39e4aa03067e100f820235 Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.153517 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17644b4_19e1_4aa9_b8d3_10e6062c5ffd.slice/crio-25a599d5849a2205b97c8e16ecd35a8cfda480b020189757144fc6a2e4d43153 WatchSource:0}: Error finding container 25a599d5849a2205b97c8e16ecd35a8cfda480b020189757144fc6a2e4d43153: Status 404 returned error can't find the container with id 25a599d5849a2205b97c8e16ecd35a8cfda480b020189757144fc6a2e4d43153 Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.171124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rvqps" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.180212 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb51bc0_a73e_40cd_bcfc_ce9b4d9ccb43.slice/crio-4b40d80c14f10de095890ef6469d01242d0954e1bf711a9a569eef40422d9005 WatchSource:0}: Error finding container 4b40d80c14f10de095890ef6469d01242d0954e1bf711a9a569eef40422d9005: Status 404 returned error can't find the container with id 4b40d80c14f10de095890ef6469d01242d0954e1bf711a9a569eef40422d9005 Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.193911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.206441 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f78bfa_f2ce_48a5_a34b_16c4db079895.slice/crio-069e63722cd7b49a8394a15f92f60d9284cb368f187c3d203f49d9ede033338d WatchSource:0}: Error finding container 069e63722cd7b49a8394a15f92f60d9284cb368f187c3d203f49d9ede033338d: Status 404 returned error can't find the container with id 069e63722cd7b49a8394a15f92f60d9284cb368f187c3d203f49d9ede033338d Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.209647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w2p7r" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.219653 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52800f2_1350_49d4_ad60_df31b5d00640.slice/crio-7935a5969cf678e73da03ce5f24bb2b1e0338dd6f8fa51eb0b890afb8036d95d WatchSource:0}: Error finding container 7935a5969cf678e73da03ce5f24bb2b1e0338dd6f8fa51eb0b890afb8036d95d: Status 404 returned error can't find the container with id 7935a5969cf678e73da03ce5f24bb2b1e0338dd6f8fa51eb0b890afb8036d95d Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.231784 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6pcwn" Jul 14 19:46:00.235929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.235848 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" Jul 14 19:46:00.454223 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.243536 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aaae312_ec5a_4cb3_8f65_700e7e980d45.slice/crio-ac32c19d6afd14f56721b26e4e2f23b286898b4dc5c5af89355e0926b30d7b86 WatchSource:0}: Error finding container ac32c19d6afd14f56721b26e4e2f23b286898b4dc5c5af89355e0926b30d7b86: Status 404 returned error can't find the container with id ac32c19d6afd14f56721b26e4e2f23b286898b4dc5c5af89355e0926b30d7b86 Jul 14 19:46:00.454223 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:00.246253 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbce3123_7485_48fa_b9fd_3b3e1e43a9ee.slice/crio-498b24376f42cf2310388bd108ad4a0415af70db1659968fdff4fb8b5c284419 WatchSource:0}: Error finding container 498b24376f42cf2310388bd108ad4a0415af70db1659968fdff4fb8b5c284419: Status 404 returned error can't find the container with id 498b24376f42cf2310388bd108ad4a0415af70db1659968fdff4fb8b5c284419 Jul 14 19:46:00.454223 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.248054 2578 kuberuntime_manager.go:1459] "Unhandled Error" err="init container egress-router-binary-copy start failed in pod multus-additional-cni-plugins-gb6f2_openshift-multus(fbce3123-7485-48fa-b9fd-3b3e1e43a9ee): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 14 19:46:00.454223 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.248095 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" podUID="fbce3123-7485-48fa-b9fd-3b3e1e43a9ee" Jul 14 19:46:00.454223 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.294300 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.483292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.483365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.483466 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.483470 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.483530 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret podName:060c531f-952a-432e-a4a3-c08cd392bdcd nodeName:}" failed. No retries permitted until 2026-07-14 19:46:01.483512427 +0000 UTC m=+3.131075975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret") pod "global-pull-secret-syncer-qzk58" (UID: "060c531f-952a-432e-a4a3-c08cd392bdcd") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.483549 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs podName:8ff66e5b-b8e1-43f9-bdba-b82de3668ef3 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:01.483539888 +0000 UTC m=+3.131103436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs") pod "network-metrics-daemon-mktvj" (UID: "8ff66e5b-b8e1-43f9-bdba-b82de3668ef3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.584001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.584201 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.584225 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.584234 2578 projected.go:196] Error preparing data for projected volume kube-api-access-w2f2m for pod openshift-network-diagnostics/network-check-target-f4clt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:00.628136 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.584302 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m podName:c0cf5e05-c1b6-4453-aea9-7f5629f80f4c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:01.584281883 +0000 UTC m=+3.231845430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2f2m" (UniqueName: "kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m") pod "network-check-target-f4clt" (UID: "c0cf5e05-c1b6-4453-aea9-7f5629f80f4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:00.912391 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.881073 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:00.912391 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.881269 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:00.912391 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.882418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:00.912391 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.882581 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:00.912391 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.891744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rvqps" event={"ID":"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43","Type":"ContainerStarted","Data":"4b40d80c14f10de095890ef6469d01242d0954e1bf711a9a569eef40422d9005"} Jul 14 19:46:00.912391 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.895932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-65rs9" event={"ID":"d17644b4-19e1-4aa9-b8d3-10e6062c5ffd","Type":"ContainerStarted","Data":"25a599d5849a2205b97c8e16ecd35a8cfda480b020189757144fc6a2e4d43153"} Jul 14 19:46:00.912391 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.899093 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerStarted","Data":"498b24376f42cf2310388bd108ad4a0415af70db1659968fdff4fb8b5c284419"} Jul 14 19:46:00.912391 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:00.907415 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"egress-router-binary-copy\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:852e5ae3f77e551118afa777a96222c9a7806f35ebea299f3e51e6fb499967c4\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" podUID="fbce3123-7485-48fa-b9fd-3b3e1e43a9ee" Jul 14 19:46:00.915223 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.912589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w2p7r" event={"ID":"a52800f2-1350-49d4-ad60-df31b5d00640","Type":"ContainerStarted","Data":"7935a5969cf678e73da03ce5f24bb2b1e0338dd6f8fa51eb0b890afb8036d95d"} Jul 14 19:46:00.915223 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.914919 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" event={"ID":"95f78bfa-f2ce-48a5-a34b-16c4db079895","Type":"ContainerStarted","Data":"069e63722cd7b49a8394a15f92f60d9284cb368f187c3d203f49d9ede033338d"} Jul 14 19:46:00.918539 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.916308 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" event={"ID":"9d525fe8-c0cb-44b9-b406-478e58125c9a","Type":"ContainerStarted","Data":"0320639a3276a6eae13ee467fe91337f701315631e39e4aa03067e100f820235"} Jul 14 19:46:00.925457 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.923458 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xthjn" event={"ID":"18ce272c-d523-4253-bf80-92b16152b6e5","Type":"ContainerStarted","Data":"af43d13d486d9378a1d934749e6cf6a8c23623f30e13efdbe975a3c69060bcaa"} Jul 14 19:46:00.927682 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.925608 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"77ada98ee0dfe7b071c3fa057655a362048753cb590fbd7a3c177a85eb2d03ef"} Jul 14 19:46:00.927682 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.927328 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" event={"ID":"2b72e59c0d82e73db7c035d21a4e6cad","Type":"ContainerStarted","Data":"b6998147c1a4de84a47f082cd7015292c5a3f820957281cc19fd4a48a1d3a94b"} Jul 14 19:46:00.943154 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.930091 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" event={"ID":"f9010125a2cabcaaf81321ade9e08441","Type":"ContainerStarted","Data":"a43ed94eee4e1359b2885a5dcdfba877de25bee60f51f71a74476f567fe54f3b"} Jul 14 19:46:00.943154 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.934854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6pcwn" event={"ID":"9aaae312-ec5a-4cb3-8f65-700e7e980d45","Type":"ContainerStarted","Data":"ac32c19d6afd14f56721b26e4e2f23b286898b4dc5c5af89355e0926b30d7b86"} Jul 14 19:46:01.046259 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.954863 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:41:00 +0000 UTC" deadline="2027-12-25 09:10:45.134093334 +0000 UTC" Jul 14 19:46:01.046259 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:00.954931 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12685h24m44.179166761s" Jul 14 19:46:01.134271 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:01.069218 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:01.493244 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:01.492370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:01.493244 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:01.492439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:01.493244 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.492607 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:01.493244 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.492673 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs podName:8ff66e5b-b8e1-43f9-bdba-b82de3668ef3 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:03.492653932 +0000 UTC m=+5.140217481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs") pod "network-metrics-daemon-mktvj" (UID: "8ff66e5b-b8e1-43f9-bdba-b82de3668ef3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:01.493244 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.493099 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:01.493244 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.493154 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret podName:060c531f-952a-432e-a4a3-c08cd392bdcd nodeName:}" failed. No retries permitted until 2026-07-14 19:46:03.493137328 +0000 UTC m=+5.140700874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret") pod "global-pull-secret-syncer-qzk58" (UID: "060c531f-952a-432e-a4a3-c08cd392bdcd") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:01.593268 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:01.593228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:01.593486 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.593468 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:01.593570 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.593496 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:01.593570 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.593510 2578 projected.go:196] Error preparing data for projected volume kube-api-access-w2f2m for pod openshift-network-diagnostics/network-check-target-f4clt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:01.593681 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.593573 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m podName:c0cf5e05-c1b6-4453-aea9-7f5629f80f4c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:03.593554008 +0000 UTC m=+5.241117555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2f2m" (UniqueName: "kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m") pod "network-check-target-f4clt" (UID: "c0cf5e05-c1b6-4453-aea9-7f5629f80f4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:01.878337 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:01.877604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:01.878337 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:01.877746 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:01.955940 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:01.955861 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 19:41:00 +0000 UTC" deadline="2028-03-14 19:48:19.811409241 +0000 UTC" Jul 14 19:46:01.955940 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:01.955903 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14616h2m17.855509616s" Jul 14 19:46:02.245844 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:02.245761 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 19:46:02.879158 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:02.879092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:02.879373 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:02.879262 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:02.879623 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:02.879093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:02.879745 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:02.879712 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:03.511338 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:03.511292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:03.511837 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:03.511377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:03.511837 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.511419 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:03.511837 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.511512 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret podName:060c531f-952a-432e-a4a3-c08cd392bdcd nodeName:}" failed. No retries permitted until 2026-07-14 19:46:07.511490884 +0000 UTC m=+9.159054431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret") pod "global-pull-secret-syncer-qzk58" (UID: "060c531f-952a-432e-a4a3-c08cd392bdcd") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:03.511837 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.511514 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:03.511837 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.511577 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs podName:8ff66e5b-b8e1-43f9-bdba-b82de3668ef3 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:07.511560031 +0000 UTC m=+9.159123591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs") pod "network-metrics-daemon-mktvj" (UID: "8ff66e5b-b8e1-43f9-bdba-b82de3668ef3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:03.612268 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:03.612228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:03.612918 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.612434 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:03.612918 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.612459 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:03.612918 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.612473 2578 projected.go:196] Error preparing data for projected volume kube-api-access-w2f2m for pod openshift-network-diagnostics/network-check-target-f4clt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:03.612918 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.612536 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m podName:c0cf5e05-c1b6-4453-aea9-7f5629f80f4c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:07.612518935 +0000 UTC m=+9.260082480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2f2m" (UniqueName: "kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m") pod "network-check-target-f4clt" (UID: "c0cf5e05-c1b6-4453-aea9-7f5629f80f4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:03.878292 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:03.878215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:03.878462 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:03.878350 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:04.878503 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:04.878003 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:04.878503 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:04.878032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:04.878503 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:04.878121 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:04.878503 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:04.878274 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:05.877960 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:05.877911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:05.878127 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:05.878070 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:06.878467 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:06.878431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:06.878996 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:06.878431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:06.878996 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:06.878570 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:06.878996 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:06.878663 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:07.547630 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:07.547593 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:07.547790 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:07.547699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:07.547875 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.547812 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:07.548025 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.547931 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs podName:8ff66e5b-b8e1-43f9-bdba-b82de3668ef3 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:15.547906797 +0000 UTC m=+17.195470348 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs") pod "network-metrics-daemon-mktvj" (UID: "8ff66e5b-b8e1-43f9-bdba-b82de3668ef3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:07.548025 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.547823 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:07.548158 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.548049 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret podName:060c531f-952a-432e-a4a3-c08cd392bdcd nodeName:}" failed. No retries permitted until 2026-07-14 19:46:15.548034039 +0000 UTC m=+17.195597588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret") pod "global-pull-secret-syncer-qzk58" (UID: "060c531f-952a-432e-a4a3-c08cd392bdcd") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:07.648791 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:07.648695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:07.649021 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.648995 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:07.649137 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.649030 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:07.649137 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.649046 2578 projected.go:196] Error preparing data for projected volume kube-api-access-w2f2m for pod openshift-network-diagnostics/network-check-target-f4clt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:07.649137 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.649108 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m podName:c0cf5e05-c1b6-4453-aea9-7f5629f80f4c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:15.649087864 +0000 UTC m=+17.296651412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2f2m" (UniqueName: "kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m") pod "network-check-target-f4clt" (UID: "c0cf5e05-c1b6-4453-aea9-7f5629f80f4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:07.877631 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:07.877579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:07.877802 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:07.877718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:08.878928 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:08.878561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:08.878928 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:08.878689 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:08.880034 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:08.879872 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:08.880034 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:08.879998 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:09.878013 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:09.877971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:09.878201 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:09.878105 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:10.877306 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:10.877263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:10.877753 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:10.877372 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:10.877753 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:10.877422 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:10.877753 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:10.877521 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:11.878150 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:11.878112 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:11.878621 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:11.878238 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:12.878285 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:12.878244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:12.878724 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:12.878300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:12.878724 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:12.878381 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:12.878724 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:12.878516 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:13.878194 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:13.878159 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:13.878382 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:13.878280 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:14.878358 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:14.878323 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:14.878525 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:14.878365 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:14.878525 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:14.878454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:14.878880 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:14.878570 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:15.607946 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:15.607910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:15.608128 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:15.607965 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:15.608128 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.608080 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:15.608128 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.608082 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:15.608277 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.608148 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs podName:8ff66e5b-b8e1-43f9-bdba-b82de3668ef3 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:31.608128239 +0000 UTC m=+33.255691783 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs") pod "network-metrics-daemon-mktvj" (UID: "8ff66e5b-b8e1-43f9-bdba-b82de3668ef3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:15.608277 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.608168 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret podName:060c531f-952a-432e-a4a3-c08cd392bdcd nodeName:}" failed. No retries permitted until 2026-07-14 19:46:31.608158723 +0000 UTC m=+33.255722462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret") pod "global-pull-secret-syncer-qzk58" (UID: "060c531f-952a-432e-a4a3-c08cd392bdcd") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:15.709150 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:15.709118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:15.709349 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.709308 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:15.709349 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.709334 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:15.709349 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.709347 2578 projected.go:196] Error preparing data for projected volume kube-api-access-w2f2m for pod openshift-network-diagnostics/network-check-target-f4clt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:15.709485 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.709412 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m podName:c0cf5e05-c1b6-4453-aea9-7f5629f80f4c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:31.709393235 +0000 UTC m=+33.356956796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2f2m" (UniqueName: "kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m") pod "network-check-target-f4clt" (UID: "c0cf5e05-c1b6-4453-aea9-7f5629f80f4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:15.877886 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:15.877792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:15.878173 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:15.878145 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:16.880589 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:16.880347 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:16.881029 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:16.880411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:16.881029 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:16.880684 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:16.881029 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:16.880749 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:17.878323 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:17.878286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:17.878506 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:17.878419 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:18.879017 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:18.878632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:18.879017 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:18.878735 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:18.879017 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:18.878811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:18.879017 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:18.878951 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:19.878063 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:19.878032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:19.878222 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:19.878132 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:20.877722 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.877696 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:20.877722 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.877711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:20.878173 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:20.877839 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:20.878173 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:20.878026 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:20.982181 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.982149 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" event={"ID":"9d525fe8-c0cb-44b9-b406-478e58125c9a","Type":"ContainerStarted","Data":"43060567717fdf61bb23e65cc150e9164867bb05899c38a4c23e7f243f943178"} Jul 14 19:46:20.986027 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.985885 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:46:20.986422 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.986377 2578 generic.go:356] "Generic (PLEG): container finished" podID="fb74204e-abe8-45da-8bbd-97eb68965a34" containerID="3d0a448392b4dd704537bef8c9c76982c274beccccfe05d3fa26a23ae7c77c1c" exitCode=1 Jul 14 19:46:20.986517 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.986456 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"a04348748905a434dc5c6b6b5556c27f991a5615d00bf3ababcba5b55e5608ed"} Jul 14 19:46:20.986578 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.986527 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerDied","Data":"3d0a448392b4dd704537bef8c9c76982c274beccccfe05d3fa26a23ae7c77c1c"} Jul 14 19:46:20.986631 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.986582 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"4ad4cc06d053a43045bd9ed75f932354c79f0479fd01f75c5c85077fca7fe1fa"} Jul 14 19:46:20.988226 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.988182 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" event={"ID":"f9010125a2cabcaaf81321ade9e08441","Type":"ContainerStarted","Data":"b727fd0c402af293849a14dfea5311fa4d444793073be0e4a467b10e83420135"} Jul 14 19:46:20.989719 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:20.989698 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6pcwn" event={"ID":"9aaae312-ec5a-4cb3-8f65-700e7e980d45","Type":"ContainerStarted","Data":"269d69c5b4776f1e12f7ff697102f0b443abe454e67c774c18ac80db66943ef3"} Jul 14 19:46:21.016533 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:21.016256 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-243.ec2.internal" podStartSLOduration=21.016235457 podStartE2EDuration="21.016235457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:21.015651718 +0000 UTC m=+22.663215284" watchObservedRunningTime="2026-07-14 19:46:21.016235457 +0000 UTC m=+22.663799023" Jul 14 19:46:21.016669 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:21.016609 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9dnkx" podStartSLOduration=1.881024564 podStartE2EDuration="22.016574847s" totalImagesPullingTime="20.135550283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:21.00211268 +0000 UTC m=+22.649676261" watchObservedRunningTime="2026-07-14 19:46:21.016574847 +0000 UTC m=+22.664138414" Jul 14 19:46:21.878168 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:21.878000 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:21.878877 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:21.878262 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:21.993542 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:21.993468 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rvqps" event={"ID":"5fb51bc0-a73e-40cd-bcfc-ce9b4d9ccb43","Type":"ContainerStarted","Data":"6f4ed8424b613581a88397b051b2ef64b31ced804c9e33ef23495846a282f887"} Jul 14 19:46:21.995173 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:21.995142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-65rs9" event={"ID":"d17644b4-19e1-4aa9-b8d3-10e6062c5ffd","Type":"ContainerStarted","Data":"0a6e5c1ca3f7eebe67a83af36b19c19a281c8ad1081c0769b67a75fd4f85c5d2"} Jul 14 19:46:21.996935 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:21.996904 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w2p7r" event={"ID":"a52800f2-1350-49d4-ad60-df31b5d00640","Type":"ContainerStarted","Data":"bc46c0d614e43c727e9186c69dbaad285fea4b7496ce5376cc239166798bff85"} Jul 14 19:46:21.998468 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:21.998447 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" event={"ID":"95f78bfa-f2ce-48a5-a34b-16c4db079895","Type":"ContainerStarted","Data":"dc566f0c8ea29fefdf92dc5d33a6c8f31bb0602c78aee45fa652d4446e1ca193"} Jul 14 19:46:21.999903 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:21.999859 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xthjn" event={"ID":"18ce272c-d523-4253-bf80-92b16152b6e5","Type":"ContainerStarted","Data":"e6affa135a47f2500b5158723b582696c36e24f7cdbe5e5c78f2e153648ddfef"} Jul 14 19:46:22.003246 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.003228 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:46:22.003662 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.003636 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"2b4b6069fe15abf2016ed0e4f771a5ec4342b9648fedfa3cd33485021609d850"} Jul 14 19:46:22.003750 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.003715 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"5e58482aa7b17c20489138d272e1870ea11178d3f152f0e4b18f8b3a9d507add"} Jul 14 19:46:22.003811 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.003762 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"ca58ba966b66d660d0cfdf967ddc3c1e29514ea111d6a943087776b64a45ae03"} Jul 14 19:46:22.005532 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.005369 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b72e59c0d82e73db7c035d21a4e6cad" containerID="aa98438ef9dc613802e97ba07daf4ed2d22201f1aab7a6608178d0637ebead87" exitCode=0 Jul 14 19:46:22.005532 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.005482 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" event={"ID":"2b72e59c0d82e73db7c035d21a4e6cad","Type":"ContainerDied","Data":"aa98438ef9dc613802e97ba07daf4ed2d22201f1aab7a6608178d0637ebead87"} Jul 14 19:46:22.010327 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.010294 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6pcwn" podStartSLOduration=1.926446094 podStartE2EDuration="22.010281932s" totalImagesPullingTime="20.083835838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:21.044737659 +0000 UTC m=+22.692301225" watchObservedRunningTime="2026-07-14 19:46:22.010281932 +0000 UTC m=+23.657845497" Jul 14 19:46:22.026553 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.026506 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rvqps" podStartSLOduration=1.942630737 podStartE2EDuration="22.026487896s" totalImagesPullingTime="20.083857159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:22.010211029 +0000 UTC m=+23.657774593" watchObservedRunningTime="2026-07-14 19:46:22.026487896 +0000 UTC m=+23.674051460" Jul 14 19:46:22.042743 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.042691 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xthjn" podStartSLOduration=2.87827406 podStartE2EDuration="23.042676074s" totalImagesPullingTime="20.164402014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:22.041957008 +0000 UTC m=+23.689520576" watchObservedRunningTime="2026-07-14 19:46:22.042676074 +0000 UTC m=+23.690239639" Jul 14 19:46:22.076352 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.076296 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w2p7r" podStartSLOduration=2.053307643 podStartE2EDuration="22.076275275s" totalImagesPullingTime="20.022967632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:22.076239417 +0000 UTC m=+23.723802982" watchObservedRunningTime="2026-07-14 19:46:22.076275275 +0000 UTC m=+23.723838842" Jul 14 19:46:22.076750 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.076683 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-65rs9" podStartSLOduration=2.96652792 podStartE2EDuration="23.076671437s" totalImagesPullingTime="20.110143517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:22.060098919 +0000 UTC m=+23.707662490" watchObservedRunningTime="2026-07-14 19:46:22.076671437 +0000 UTC m=+23.724235003" Jul 14 19:46:22.722080 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.722046 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 19:46:22.876225 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.876038 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T19:46:22.722070142Z","UUID":"a1c37ba2-b8c0-4550-b3ab-8fb1b2680df9","Handler":null,"Name":"","Endpoint":""} Jul 14 19:46:22.877256 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.877232 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:22.877335 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.877293 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:22.877411 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:22.877389 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:22.877516 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:22.877492 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:22.877621 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.877609 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 19:46:22.877690 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:22.877646 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 19:46:23.008880 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:23.008823 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" event={"ID":"2b72e59c0d82e73db7c035d21a4e6cad","Type":"ContainerStarted","Data":"c8cf603bc774ed31fdba2e4f51fb968fb93aa546a569be9e0bebc4579dd96551"} Jul 14 19:46:23.010260 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:23.010238 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbce3123-7485-48fa-b9fd-3b3e1e43a9ee" containerID="0fe728e50ac1502003d89e460a02f2aeddde231d29b846899a665b8e288511e3" exitCode=0 Jul 14 19:46:23.010374 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:23.010306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerDied","Data":"0fe728e50ac1502003d89e460a02f2aeddde231d29b846899a665b8e288511e3"} Jul 14 19:46:23.011841 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:23.011809 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" event={"ID":"95f78bfa-f2ce-48a5-a34b-16c4db079895","Type":"ContainerStarted","Data":"543f3520d220ae5425d6e8c4352683a7def4d136e518f602e40e21c12a030758"} Jul 14 19:46:23.048165 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:23.048126 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-243.ec2.internal" podStartSLOduration=23.048111757 podStartE2EDuration="23.048111757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:23.026014012 +0000 UTC m=+24.673577578" watchObservedRunningTime="2026-07-14 19:46:23.048111757 +0000 UTC m=+24.695675321" Jul 14 19:46:23.878192 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:23.878161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:23.878420 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:23.878266 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:24.017216 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:24.017187 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:46:24.017640 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:24.017616 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"de4495dc7a20224ab868355eab37f657fc14bca99b79ce95a43671946a11bc3b"} Jul 14 19:46:24.878060 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:24.877901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:24.878178 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:24.877900 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:24.878178 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:24.878144 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:24.878308 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:24.878286 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:25.021820 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:25.021784 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" event={"ID":"95f78bfa-f2ce-48a5-a34b-16c4db079895","Type":"ContainerStarted","Data":"33263e1c89b402991a2e03a1452d182c0c34a2f909cb178d30449141597feee2"} Jul 14 19:46:25.040573 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:25.040526 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s89ks" podStartSLOduration=1.340526869 podStartE2EDuration="25.040507277s" totalImagesPullingTime="23.699980408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:25.040365127 +0000 UTC m=+26.687928691" watchObservedRunningTime="2026-07-14 19:46:25.040507277 +0000 UTC m=+26.688070842" Jul 14 19:46:25.509377 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:25.509339 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:25.510022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:25.509994 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:25.877538 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:25.877502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:25.877732 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:25.877643 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:26.024390 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:26.024348 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:26.025100 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:26.025067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-65rs9" Jul 14 19:46:26.877622 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:26.877367 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:26.877797 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:26.877450 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:26.877797 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:26.877736 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:26.877920 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:26.877823 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:27.030653 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:27.030624 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:46:27.031469 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:27.031049 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"c600851cc75ca42d78b56edb5ad0f35372e7f8f5d105aa1f4ca75014ebd8cca1"} Jul 14 19:46:27.031581 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:27.031562 2578 scope.go:117] "RemoveContainer" containerID="3d0a448392b4dd704537bef8c9c76982c274beccccfe05d3fa26a23ae7c77c1c" Jul 14 19:46:27.877641 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:27.877611 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:27.877909 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:27.877749 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:28.037902 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.037865 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:46:28.038360 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.038326 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" event={"ID":"fb74204e-abe8-45da-8bbd-97eb68965a34","Type":"ContainerStarted","Data":"9babfcf7163bd2c0288421f52b9e10f617867aef79fe1d86c38485e38b36ec2c"} Jul 14 19:46:28.038527 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.038512 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:28.038750 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.038727 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:28.038870 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.038805 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:28.064407 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.064381 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:28.064608 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.064518 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:46:28.069099 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.069052 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" podStartSLOduration=8.543860873 podStartE2EDuration="29.069032496s" totalImagesPullingTime="20.525171623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:45:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:28.068536343 +0000 UTC m=+29.716099909" watchObservedRunningTime="2026-07-14 19:46:28.069032496 +0000 UTC m=+29.716596063" Jul 14 19:46:28.159116 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.159026 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f4clt"] Jul 14 19:46:28.159279 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.159172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:28.159321 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:28.159278 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:28.162086 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.162056 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qzk58"] Jul 14 19:46:28.162221 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.162176 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:28.162301 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:28.162277 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:28.162773 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.162748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mktvj"] Jul 14 19:46:28.162994 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:28.162891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:28.163071 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:28.162996 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:29.878063 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:29.878033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:29.878466 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:29.878080 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:29.878466 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:29.878173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:29.878466 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:29.878187 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:29.878466 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:29.878280 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:29.878466 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:29.878351 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:30.046650 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:30.046612 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbce3123-7485-48fa-b9fd-3b3e1e43a9ee" containerID="3692cf9714685491051ac653dc4d8ab74f143f7948379badeb37f54ec5cba8a1" exitCode=0 Jul 14 19:46:30.046819 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:30.046680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerDied","Data":"3692cf9714685491051ac653dc4d8ab74f143f7948379badeb37f54ec5cba8a1"} Jul 14 19:46:31.051538 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:31.051374 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerStarted","Data":"ed38016fd6ba00df9f9c414fb220ebf6d5a59b8e862e986fad9a5541972058bc"} Jul 14 19:46:31.633412 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:31.633382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:31.633562 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:31.633426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:31.633562 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.633534 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:31.633562 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.633538 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:31.633663 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.633587 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs podName:8ff66e5b-b8e1-43f9-bdba-b82de3668ef3 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:03.633572343 +0000 UTC m=+65.281135887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs") pod "network-metrics-daemon-mktvj" (UID: "8ff66e5b-b8e1-43f9-bdba-b82de3668ef3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 19:46:31.633663 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.633599 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret podName:060c531f-952a-432e-a4a3-c08cd392bdcd nodeName:}" failed. No retries permitted until 2026-07-14 19:47:03.633593487 +0000 UTC m=+65.281157031 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret") pod "global-pull-secret-syncer-qzk58" (UID: "060c531f-952a-432e-a4a3-c08cd392bdcd") : object "kube-system"/"original-pull-secret" not registered Jul 14 19:46:31.734160 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:31.734124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:31.734305 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.734287 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 19:46:31.734357 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.734307 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 19:46:31.734357 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.734319 2578 projected.go:196] Error preparing data for projected volume kube-api-access-w2f2m for pod openshift-network-diagnostics/network-check-target-f4clt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:31.734418 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.734371 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m podName:c0cf5e05-c1b6-4453-aea9-7f5629f80f4c nodeName:}" failed. No retries permitted until 2026-07-14 19:47:03.734357426 +0000 UTC m=+65.381920972 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w2f2m" (UniqueName: "kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m") pod "network-check-target-f4clt" (UID: "c0cf5e05-c1b6-4453-aea9-7f5629f80f4c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 19:46:31.877756 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:31.877722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:31.877938 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:31.877821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:31.877938 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:31.877842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:31.877938 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.877921 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:31.878037 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.877983 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:31.878037 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:31.877839 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:32.057342 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:32.057311 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbce3123-7485-48fa-b9fd-3b3e1e43a9ee" containerID="ed38016fd6ba00df9f9c414fb220ebf6d5a59b8e862e986fad9a5541972058bc" exitCode=0 Jul 14 19:46:32.057757 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:32.057355 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerDied","Data":"ed38016fd6ba00df9f9c414fb220ebf6d5a59b8e862e986fad9a5541972058bc"} Jul 14 19:46:33.877754 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:33.877721 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:33.878145 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:33.877758 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:33.878145 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:33.877731 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:33.878145 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:33.877852 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f4clt" podUID="c0cf5e05-c1b6-4453-aea9-7f5629f80f4c" Jul 14 19:46:33.878145 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:33.877913 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mktvj" podUID="8ff66e5b-b8e1-43f9-bdba-b82de3668ef3" Jul 14 19:46:33.878145 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:33.877976 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qzk58" podUID="060c531f-952a-432e-a4a3-c08cd392bdcd" Jul 14 19:46:34.064391 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.064362 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbce3123-7485-48fa-b9fd-3b3e1e43a9ee" containerID="4a2bad15619523781d6e9144eee7ff98a07c941487beb414d2f3faa85afac169" exitCode=0 Jul 14 19:46:34.064533 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.064405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerDied","Data":"4a2bad15619523781d6e9144eee7ff98a07c941487beb414d2f3faa85afac169"} Jul 14 19:46:34.194424 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.194354 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-243.ec2.internal" event="NodeReady" Jul 14 19:46:34.194620 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.194482 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 19:46:34.246681 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.246648 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64cdb65b65-wgdv9"] Jul 14 19:46:34.249122 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.249106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.251773 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.251751 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 19:46:34.252248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.252227 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 19:46:34.252362 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.252258 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4vjs\"" Jul 14 19:46:34.252362 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.252333 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 19:46:34.256612 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.256591 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 19:46:34.261703 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.259907 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4"] Jul 14 19:46:34.263819 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.263797 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lb7wt"] Jul 14 19:46:34.264059 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.263930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4" Jul 14 19:46:34.266605 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.266582 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb"] Jul 14 19:46:34.266748 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.266684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.267053 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.267027 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dsp4q\"" Jul 14 19:46:34.267053 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.267048 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.267270 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.267213 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.270299 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.270280 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-h7mpd"] Jul 14 19:46:34.270396 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.270375 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.271412 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.271396 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 19:46:34.271757 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.271745 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 19:46:34.271801 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.271758 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7w2mk\"" Jul 14 19:46:34.272854 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.272817 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.272990 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.272973 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv"] Jul 14 19:46:34.273123 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.273101 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.273294 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.273251 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.273757 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.273739 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n9mmf\"" Jul 14 19:46:34.273862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.273740 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 14 19:46:34.273862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.273743 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 14 19:46:34.275859 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.275842 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk"] Jul 14 19:46:34.275934 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.275900 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.276703 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.276665 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.276802 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.276691 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 14 19:46:34.277034 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.276999 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 14 19:46:34.277111 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.277043 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.277111 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.277103 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tsrxn\"" Jul 14 19:46:34.278390 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.278371 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p876g\"" Jul 14 19:46:34.278553 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.278539 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv"] Jul 14 19:46:34.278653 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.278638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:34.278726 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.278672 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.278780 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.278770 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 14 19:46:34.279189 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.279173 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 14 19:46:34.279476 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.279463 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.281226 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.281207 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 14 19:46:34.281311 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.281238 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6wwg6"] Jul 14 19:46:34.281311 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.281259 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.281311 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.281268 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-rzxh5\"" Jul 14 19:46:34.281460 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.281443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.281460 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.281457 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.282799 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.282779 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 14 19:46:34.283980 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.283946 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lhn87"] Jul 14 19:46:34.284412 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.284387 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 14 19:46:34.284556 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.284425 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.284620 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.284542 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.284620 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.284578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.284620 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.284553 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rv76s\"" Jul 14 19:46:34.284620 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.284435 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 14 19:46:34.286685 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.286667 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b7cc8dd66-pmcdg"] Jul 14 19:46:34.286897 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.286882 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lhn87" Jul 14 19:46:34.289902 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.288073 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.289902 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.288398 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z5fb7\"" Jul 14 19:46:34.289902 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.288948 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 14 19:46:34.289902 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.289140 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 14 19:46:34.289902 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.289421 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.289902 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.289785 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.290208 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.290011 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 19:46:34.292303 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.291505 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-46xb9\"" Jul 14 19:46:34.292849 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.292801 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.297326 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.297284 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cdb65b65-wgdv9"] Jul 14 19:46:34.297430 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.297343 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4"] Jul 14 19:46:34.297430 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.297383 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb"] Jul 14 19:46:34.297430 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.297417 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv"] Jul 14 19:46:34.297715 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.297695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.299348 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.299301 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lb7wt"] Jul 14 19:46:34.301190 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.301119 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6wwg6"] Jul 14 19:46:34.301459 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.301437 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 14 19:46:34.301683 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.301446 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hfm59\"" Jul 14 19:46:34.301766 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.301749 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 14 19:46:34.302258 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.302238 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 19:46:34.302473 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.302456 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 19:46:34.302628 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.302608 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 14 19:46:34.302697 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.302646 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk"] Jul 14 19:46:34.302697 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.302675 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 14 19:46:34.303446 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.303422 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv"] Jul 14 19:46:34.304799 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.304781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 14 19:46:34.305500 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.305478 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-h7mpd"] Jul 14 19:46:34.306460 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.306440 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b7cc8dd66-pmcdg"] Jul 14 19:46:34.307164 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.307148 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lhn87"] Jul 14 19:46:34.353108 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.353080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.353238 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.353134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-kube-api-access-cpq4q\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.353238 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.353191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-image-registry-private-configuration\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.353238 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.353217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77be464-74ee-490f-aed1-99bed17482ef-ca-trust-extracted\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.353238 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.353236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-installation-pull-secrets\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.353384 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.353251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-bound-sa-token\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.353384 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.353293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-registry-certificates\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.353384 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.353318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-trusted-ca\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.454062 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.453979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-image-registry-private-configuration\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.454062 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454014 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzcc\" (UniqueName: \"kubernetes.io/projected/efee90be-3aa3-43a7-9a09-9a537528cff0-kube-api-access-tvzcc\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.454062 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.454062 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-serving-cert\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.454062 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454065 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbh5\" (UniqueName: \"kubernetes.io/projected/3dc21d0b-1485-4386-8f3b-ebdef6134092-kube-api-access-pvbh5\") pod \"ingress-canary-lhn87\" (UID: \"3dc21d0b-1485-4386-8f3b-ebdef6134092\") " pod="openshift-ingress-canary/ingress-canary-lhn87" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454101 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-stats-auth\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454133 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5d04d06d-c61d-4982-895f-50add148475a-serving-cert\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454229 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c335c34-95b3-4b24-8b1c-0e503876aeb9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.454348 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.454366 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdb65b65-wgdv9: secret "image-registry-tls" not found Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d04d06d-c61d-4982-895f-50add148475a-config\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg9g\" (UniqueName: \"kubernetes.io/projected/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-kube-api-access-shg9g\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.454419 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.454423 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls podName:c77be464-74ee-490f-aed1-99bed17482ef nodeName:}" failed. No retries permitted until 2026-07-14 19:46:34.954407082 +0000 UTC m=+36.601970628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls") pod "image-registry-64cdb65b65-wgdv9" (UID: "c77be464-74ee-490f-aed1-99bed17482ef") : secret "image-registry-tls" not found Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-config\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4p76\" (UniqueName: \"kubernetes.io/projected/4c335c34-95b3-4b24-8b1c-0e503876aeb9-kube-api-access-t4p76\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-kube-api-access-cpq4q\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxz9j\" (UniqueName: \"kubernetes.io/projected/85b2851b-b372-4935-9b8b-379379b4ae92-kube-api-access-sxz9j\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/efee90be-3aa3-43a7-9a09-9a537528cff0-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94fdd086-ee53-49c0-af48-d2602acad011-config-volume\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454610 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-default-certificate\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77be464-74ee-490f-aed1-99bed17482ef-ca-trust-extracted\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-installation-pull-secrets\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-bound-sa-token\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b2851b-b372-4935-9b8b-379379b4ae92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b2851b-b372-4935-9b8b-379379b4ae92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3dc21d0b-1485-4386-8f3b-ebdef6134092-cert\") pod \"ingress-canary-lhn87\" (UID: \"3dc21d0b-1485-4386-8f3b-ebdef6134092\") " pod="openshift-ingress-canary/ingress-canary-lhn87" Jul 14 19:46:34.454910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-trusted-ca\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-registry-certificates\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454972 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77be464-74ee-490f-aed1-99bed17482ef-ca-trust-extracted\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.454984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-trusted-ca\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx9gz\" (UniqueName: \"kubernetes.io/projected/40c179c1-a98f-437a-874a-ebb88a6b8200-kube-api-access-wx9gz\") pod \"volume-data-source-validator-78495597f6-j62t4\" (UID: \"40c179c1-a98f-437a-874a-ebb88a6b8200\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455050 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrwv\" (UniqueName: \"kubernetes.io/projected/5d04d06d-c61d-4982-895f-50add148475a-kube-api-access-sfrwv\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85b2851b-b372-4935-9b8b-379379b4ae92-serving-cert\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85b2851b-b372-4935-9b8b-379379b4ae92-tmp\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fdd086-ee53-49c0-af48-d2602acad011-tmp-dir\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455148 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455215 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmwmf\" (UniqueName: \"kubernetes.io/projected/94fdd086-ee53-49c0-af48-d2602acad011-kube-api-access-nmwmf\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85b2851b-b372-4935-9b8b-379379b4ae92-snapshots\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455298 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqzt\" (UniqueName: \"kubernetes.io/projected/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-kube-api-access-fhqzt\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c335c34-95b3-4b24-8b1c-0e503876aeb9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdlv\" (UniqueName: \"kubernetes.io/projected/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-kube-api-access-vpdlv\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:34.455534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-registry-certificates\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.456172 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.455793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-trusted-ca\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.458396 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.458375 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-image-registry-private-configuration\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.458501 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.458412 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-installation-pull-secrets\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.467814 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.467793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-bound-sa-token\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.467919 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.467864 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-kube-api-access-cpq4q\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.556284 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.556284 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/efee90be-3aa3-43a7-9a09-9a537528cff0-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.556528 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94fdd086-ee53-49c0-af48-d2602acad011-config-volume\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.556528 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-default-certificate\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.556528 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b2851b-b372-4935-9b8b-379379b4ae92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.556528 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b2851b-b372-4935-9b8b-379379b4ae92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.556528 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.556507 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:46:34.556528 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3dc21d0b-1485-4386-8f3b-ebdef6134092-cert\") pod \"ingress-canary-lhn87\" (UID: \"3dc21d0b-1485-4386-8f3b-ebdef6134092\") " pod="openshift-ingress-canary/ingress-canary-lhn87" Jul 14 19:46:34.556793 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-trusted-ca\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.556793 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.556581 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls podName:94fdd086-ee53-49c0-af48-d2602acad011 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:35.056559089 +0000 UTC m=+36.704122662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls") pod "dns-default-lb7wt" (UID: "94fdd086-ee53-49c0-af48-d2602acad011") : secret "dns-default-metrics-tls" not found Jul 14 19:46:34.556793 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9gz\" (UniqueName: \"kubernetes.io/projected/40c179c1-a98f-437a-874a-ebb88a6b8200-kube-api-access-wx9gz\") pod \"volume-data-source-validator-78495597f6-j62t4\" (UID: \"40c179c1-a98f-437a-874a-ebb88a6b8200\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4" Jul 14 19:46:34.556793 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrwv\" (UniqueName: \"kubernetes.io/projected/5d04d06d-c61d-4982-895f-50add148475a-kube-api-access-sfrwv\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.556793 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85b2851b-b372-4935-9b8b-379379b4ae92-serving-cert\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.556793 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85b2851b-b372-4935-9b8b-379379b4ae92-tmp\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.556793 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556738 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fdd086-ee53-49c0-af48-d2602acad011-tmp-dir\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.556793 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmwmf\" (UniqueName: \"kubernetes.io/projected/94fdd086-ee53-49c0-af48-d2602acad011-kube-api-access-nmwmf\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85b2851b-b372-4935-9b8b-379379b4ae92-snapshots\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqzt\" (UniqueName: \"kubernetes.io/projected/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-kube-api-access-fhqzt\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c335c34-95b3-4b24-8b1c-0e503876aeb9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.556991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdlv\" (UniqueName: \"kubernetes.io/projected/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-kube-api-access-vpdlv\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/94fdd086-ee53-49c0-af48-d2602acad011-config-volume\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzcc\" (UniqueName: \"kubernetes.io/projected/efee90be-3aa3-43a7-9a09-9a537528cff0-kube-api-access-tvzcc\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-serving-cert\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbh5\" (UniqueName: \"kubernetes.io/projected/3dc21d0b-1485-4386-8f3b-ebdef6134092-kube-api-access-pvbh5\") pod \"ingress-canary-lhn87\" (UID: \"3dc21d0b-1485-4386-8f3b-ebdef6134092\") " pod="openshift-ingress-canary/ingress-canary-lhn87" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557179 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b2851b-b372-4935-9b8b-379379b4ae92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-stats-auth\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5d04d06d-c61d-4982-895f-50add148475a-serving-cert\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557309 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/efee90be-3aa3-43a7-9a09-9a537528cff0-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.557645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c335c34-95b3-4b24-8b1c-0e503876aeb9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d04d06d-c61d-4982-895f-50add148475a-config\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg9g\" (UniqueName: \"kubernetes.io/projected/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-kube-api-access-shg9g\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557451 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94fdd086-ee53-49c0-af48-d2602acad011-tmp-dir\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-config\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4p76\" (UniqueName: \"kubernetes.io/projected/4c335c34-95b3-4b24-8b1c-0e503876aeb9-kube-api-access-t4p76\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557495 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b2851b-b372-4935-9b8b-379379b4ae92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.557604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxz9j\" (UniqueName: \"kubernetes.io/projected/85b2851b-b372-4935-9b8b-379379b4ae92-kube-api-access-sxz9j\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.558016 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.558068 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-trusted-ca\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.558079 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls podName:efee90be-3aa3-43a7-9a09-9a537528cff0 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:35.058062259 +0000 UTC m=+36.705625807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-f4mvb" (UID: "efee90be-3aa3-43a7-9a09-9a537528cff0") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:34.558495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.558333 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85b2851b-b372-4935-9b8b-379379b4ae92-snapshots\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.559131 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.558589 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 19:46:34.559131 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.558622 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4c335c34-95b3-4b24-8b1c-0e503876aeb9-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.559131 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.558636 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls podName:1adbaaac-9ef8-44c6-b06d-1927d805a0ee nodeName:}" failed. No retries permitted until 2026-07-14 19:46:35.058620236 +0000 UTC m=+36.706183780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls") pod "cluster-samples-operator-65db858894-mfjjk" (UID: "1adbaaac-9ef8-44c6-b06d-1927d805a0ee") : secret "samples-operator-tls" not found Jul 14 19:46:34.559131 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.558710 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:35.058699346 +0000 UTC m=+36.706262921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : configmap references non-existent config key: service-ca.crt Jul 14 19:46:34.559131 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.559116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85b2851b-b372-4935-9b8b-379379b4ae92-tmp\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.559384 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.559189 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:46:34.559384 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.559234 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:35.059220451 +0000 UTC m=+36.706784010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : secret "router-metrics-certs-default" not found Jul 14 19:46:34.559384 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.559296 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-config\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.559384 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.559350 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d04d06d-c61d-4982-895f-50add148475a-config\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.560702 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.560653 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3dc21d0b-1485-4386-8f3b-ebdef6134092-cert\") pod \"ingress-canary-lhn87\" (UID: \"3dc21d0b-1485-4386-8f3b-ebdef6134092\") " pod="openshift-ingress-canary/ingress-canary-lhn87" Jul 14 19:46:34.560839 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.560798 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4c335c34-95b3-4b24-8b1c-0e503876aeb9-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.560969 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.560918 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-default-certificate\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.561046 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.561023 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-stats-auth\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.561114 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.561099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-serving-cert\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.561148 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.561124 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5d04d06d-c61d-4982-895f-50add148475a-serving-cert\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.561951 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.561934 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85b2851b-b372-4935-9b8b-379379b4ae92-serving-cert\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.570317 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.570299 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzcc\" (UniqueName: \"kubernetes.io/projected/efee90be-3aa3-43a7-9a09-9a537528cff0-kube-api-access-tvzcc\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:34.572497 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.572478 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbh5\" (UniqueName: \"kubernetes.io/projected/3dc21d0b-1485-4386-8f3b-ebdef6134092-kube-api-access-pvbh5\") pod \"ingress-canary-lhn87\" (UID: \"3dc21d0b-1485-4386-8f3b-ebdef6134092\") " pod="openshift-ingress-canary/ingress-canary-lhn87" Jul 14 19:46:34.572636 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.572620 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdlv\" (UniqueName: \"kubernetes.io/projected/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-kube-api-access-vpdlv\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:34.577781 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.577757 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqzt\" (UniqueName: \"kubernetes.io/projected/371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97-kube-api-access-fhqzt\") pod \"console-operator-7866497494-h7mpd\" (UID: \"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97\") " pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.578338 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.578315 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg9g\" (UniqueName: \"kubernetes.io/projected/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-kube-api-access-shg9g\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:34.578927 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.578898 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrwv\" (UniqueName: \"kubernetes.io/projected/5d04d06d-c61d-4982-895f-50add148475a-kube-api-access-sfrwv\") pod \"service-ca-operator-7695fc6c8c-pzbmv\" (UID: \"5d04d06d-c61d-4982-895f-50add148475a\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.579059 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.579040 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx9gz\" (UniqueName: \"kubernetes.io/projected/40c179c1-a98f-437a-874a-ebb88a6b8200-kube-api-access-wx9gz\") pod \"volume-data-source-validator-78495597f6-j62t4\" (UID: \"40c179c1-a98f-437a-874a-ebb88a6b8200\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4" Jul 14 19:46:34.579216 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.579200 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxz9j\" (UniqueName: \"kubernetes.io/projected/85b2851b-b372-4935-9b8b-379379b4ae92-kube-api-access-sxz9j\") pod \"insights-operator-68d9dd94cf-6wwg6\" (UID: \"85b2851b-b372-4935-9b8b-379379b4ae92\") " pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.579784 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.579764 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4p76\" (UniqueName: \"kubernetes.io/projected/4c335c34-95b3-4b24-8b1c-0e503876aeb9-kube-api-access-t4p76\") pod \"kube-storage-version-migrator-operator-799cb7f958-795zv\" (UID: \"4c335c34-95b3-4b24-8b1c-0e503876aeb9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.579881 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.579867 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmwmf\" (UniqueName: \"kubernetes.io/projected/94fdd086-ee53-49c0-af48-d2602acad011-kube-api-access-nmwmf\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:34.605378 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.605349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:34.606693 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.606663 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:34.610735 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.610718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" Jul 14 19:46:34.622625 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.622598 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" Jul 14 19:46:34.628770 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.628741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" Jul 14 19:46:34.633819 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.633773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lhn87" Jul 14 19:46:34.706650 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.706261 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:34.806449 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.803436 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv"] Jul 14 19:46:34.815886 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.815859 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-h7mpd"] Jul 14 19:46:34.818708 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:34.818684 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371cf92f_1fd2_4277_a2b0_bd5fb0bb2f97.slice/crio-109c32726e23e60ae431c082046e7dd04ae4264529a5de6e29d539ecf133b5d2 WatchSource:0}: Error finding container 109c32726e23e60ae431c082046e7dd04ae4264529a5de6e29d539ecf133b5d2: Status 404 returned error can't find the container with id 109c32726e23e60ae431c082046e7dd04ae4264529a5de6e29d539ecf133b5d2 Jul 14 19:46:34.840049 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.840026 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-6wwg6"] Jul 14 19:46:34.841996 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:34.841974 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b2851b_b372_4935_9b8b_379379b4ae92.slice/crio-b5fe237d20bf4eee04cbc2fa22d61b1806ce573a6aa984bfb551a085cfa715e9 WatchSource:0}: Error finding container b5fe237d20bf4eee04cbc2fa22d61b1806ce573a6aa984bfb551a085cfa715e9: Status 404 returned error can't find the container with id b5fe237d20bf4eee04cbc2fa22d61b1806ce573a6aa984bfb551a085cfa715e9 Jul 14 19:46:34.844413 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.844392 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lhn87"] Jul 14 19:46:34.847098 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:34.847079 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc21d0b_1485_4386_8f3b_ebdef6134092.slice/crio-5e50826d05e5ace87c6212c935fb3c49a12626143165acf7ae077185da475a4b WatchSource:0}: Error finding container 5e50826d05e5ace87c6212c935fb3c49a12626143165acf7ae077185da475a4b: Status 404 returned error can't find the container with id 5e50826d05e5ace87c6212c935fb3c49a12626143165acf7ae077185da475a4b Jul 14 19:46:34.852353 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.852333 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv"] Jul 14 19:46:34.854936 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:34.854904 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c335c34_95b3_4b24_8b1c_0e503876aeb9.slice/crio-c38767d310b8d0c81767c273bf645cfcd3fb026fb33788514b150b4f09c1aa8f WatchSource:0}: Error finding container c38767d310b8d0c81767c273bf645cfcd3fb026fb33788514b150b4f09c1aa8f: Status 404 returned error can't find the container with id c38767d310b8d0c81767c273bf645cfcd3fb026fb33788514b150b4f09c1aa8f Jul 14 19:46:34.873971 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.873947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4" Jul 14 19:46:34.961401 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.961331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:34.962017 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.961485 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:46:34.962017 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.961504 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdb65b65-wgdv9: secret "image-registry-tls" not found Jul 14 19:46:34.962017 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:34.961570 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls podName:c77be464-74ee-490f-aed1-99bed17482ef nodeName:}" failed. No retries permitted until 2026-07-14 19:46:35.961552798 +0000 UTC m=+37.609116350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls") pod "image-registry-64cdb65b65-wgdv9" (UID: "c77be464-74ee-490f-aed1-99bed17482ef") : secret "image-registry-tls" not found Jul 14 19:46:34.999131 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:34.999102 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4"] Jul 14 19:46:35.001608 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:35.001582 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c179c1_a98f_437a_874a_ebb88a6b8200.slice/crio-574fa38bb1d6eea342d32a93093871c35d79e89571fbbb39ff51c40deb956160 WatchSource:0}: Error finding container 574fa38bb1d6eea342d32a93093871c35d79e89571fbbb39ff51c40deb956160: Status 404 returned error can't find the container with id 574fa38bb1d6eea342d32a93093871c35d79e89571fbbb39ff51c40deb956160 Jul 14 19:46:35.061987 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.061953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:35.062109 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.062001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:35.062109 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.062063 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:35.062109 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.062092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:35.062255 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062117 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:46:35.062255 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.062145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:35.062255 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062177 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:36.062157286 +0000 UTC m=+37.709720833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : secret "router-metrics-certs-default" not found Jul 14 19:46:35.062255 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062214 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:35.062255 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062249 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls podName:efee90be-3aa3-43a7-9a09-9a537528cff0 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:36.062237985 +0000 UTC m=+37.709801536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-f4mvb" (UID: "efee90be-3aa3-43a7-9a09-9a537528cff0") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:35.062460 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062263 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 19:46:35.062460 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062308 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:36.062292459 +0000 UTC m=+37.709856002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : configmap references non-existent config key: service-ca.crt Jul 14 19:46:35.062460 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062326 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls podName:1adbaaac-9ef8-44c6-b06d-1927d805a0ee nodeName:}" failed. No retries permitted until 2026-07-14 19:46:36.062320107 +0000 UTC m=+37.709883651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls") pod "cluster-samples-operator-65db858894-mfjjk" (UID: "1adbaaac-9ef8-44c6-b06d-1927d805a0ee") : secret "samples-operator-tls" not found Jul 14 19:46:35.062460 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062345 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:46:35.062460 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.062387 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls podName:94fdd086-ee53-49c0-af48-d2602acad011 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:36.062362869 +0000 UTC m=+37.709926412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls") pod "dns-default-lb7wt" (UID: "94fdd086-ee53-49c0-af48-d2602acad011") : secret "dns-default-metrics-tls" not found Jul 14 19:46:35.068149 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.068118 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" event={"ID":"85b2851b-b372-4935-9b8b-379379b4ae92","Type":"ContainerStarted","Data":"b5fe237d20bf4eee04cbc2fa22d61b1806ce573a6aa984bfb551a085cfa715e9"} Jul 14 19:46:35.069321 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.069289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" event={"ID":"5d04d06d-c61d-4982-895f-50add148475a","Type":"ContainerStarted","Data":"54077cb751b7e898ef7ba01dae38cf6293788a744afe04fc8bb3442e3f873bb5"} Jul 14 19:46:35.070396 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.070369 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-h7mpd" event={"ID":"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97","Type":"ContainerStarted","Data":"109c32726e23e60ae431c082046e7dd04ae4264529a5de6e29d539ecf133b5d2"} Jul 14 19:46:35.071382 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.071362 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4" event={"ID":"40c179c1-a98f-437a-874a-ebb88a6b8200","Type":"ContainerStarted","Data":"574fa38bb1d6eea342d32a93093871c35d79e89571fbbb39ff51c40deb956160"} Jul 14 19:46:35.072363 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.072345 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lhn87" event={"ID":"3dc21d0b-1485-4386-8f3b-ebdef6134092","Type":"ContainerStarted","Data":"5e50826d05e5ace87c6212c935fb3c49a12626143165acf7ae077185da475a4b"} Jul 14 19:46:35.073366 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.073344 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" event={"ID":"4c335c34-95b3-4b24-8b1c-0e503876aeb9","Type":"ContainerStarted","Data":"c38767d310b8d0c81767c273bf645cfcd3fb026fb33788514b150b4f09c1aa8f"} Jul 14 19:46:35.167663 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.167628 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:35.292104 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.292029 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:35.399397 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.399359 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:35.500904 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.500863 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:35.597737 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.597707 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:35.878315 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.878239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:46:35.878649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.878633 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:46:35.879011 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.878993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:46:35.882387 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.882362 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 19:46:35.882794 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.882777 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:46:35.883037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.883016 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xm9xl\"" Jul 14 19:46:35.883196 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.883181 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:46:35.883374 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.883358 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mvlkh\"" Jul 14 19:46:35.883792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.883562 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 19:46:35.941059 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.940872 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:35.972289 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:35.971682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:35.972289 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.971880 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:46:35.972289 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.971896 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdb65b65-wgdv9: secret "image-registry-tls" not found Jul 14 19:46:35.972289 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:35.971958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls podName:c77be464-74ee-490f-aed1-99bed17482ef nodeName:}" failed. No retries permitted until 2026-07-14 19:46:37.971937377 +0000 UTC m=+39.619500923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls") pod "image-registry-64cdb65b65-wgdv9" (UID: "c77be464-74ee-490f-aed1-99bed17482ef") : secret "image-registry-tls" not found Jul 14 19:46:36.010051 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:36.010015 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:36.073248 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:36.073193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:36.073468 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:36.073286 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:36.073468 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:36.073341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:36.073468 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:36.073403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:36.073468 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:36.073466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:36.073748 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.073731 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:46:36.073808 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.073798 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls podName:94fdd086-ee53-49c0-af48-d2602acad011 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:38.073778618 +0000 UTC m=+39.721342166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls") pod "dns-default-lb7wt" (UID: "94fdd086-ee53-49c0-af48-d2602acad011") : secret "dns-default-metrics-tls" not found Jul 14 19:46:36.074013 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.073996 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:46:36.074158 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.074057 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:38.074040228 +0000 UTC m=+39.721603787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : secret "router-metrics-certs-default" not found Jul 14 19:46:36.074272 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.074202 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:36.074272 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.074250 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls podName:efee90be-3aa3-43a7-9a09-9a537528cff0 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:38.074235856 +0000 UTC m=+39.721799405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-f4mvb" (UID: "efee90be-3aa3-43a7-9a09-9a537528cff0") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:36.074378 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.074313 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 19:46:36.074378 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.074347 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls podName:1adbaaac-9ef8-44c6-b06d-1927d805a0ee nodeName:}" failed. No retries permitted until 2026-07-14 19:46:38.074336009 +0000 UTC m=+39.721899557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls") pod "cluster-samples-operator-65db858894-mfjjk" (UID: "1adbaaac-9ef8-44c6-b06d-1927d805a0ee") : secret "samples-operator-tls" not found Jul 14 19:46:36.074594 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:36.074581 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:38.074568646 +0000 UTC m=+39.722132190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : configmap references non-existent config key: service-ca.crt Jul 14 19:46:36.354252 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:36.354197 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:36.457979 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:36.457688 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:37.990716 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:37.990626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:37.991175 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:37.990908 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:46:37.991175 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:37.990934 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdb65b65-wgdv9: secret "image-registry-tls" not found Jul 14 19:46:37.991175 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:37.991003 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls podName:c77be464-74ee-490f-aed1-99bed17482ef nodeName:}" failed. No retries permitted until 2026-07-14 19:46:41.990980744 +0000 UTC m=+43.638544301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls") pod "image-registry-64cdb65b65-wgdv9" (UID: "c77be464-74ee-490f-aed1-99bed17482ef") : secret "image-registry-tls" not found Jul 14 19:46:38.092095 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.092061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:38.092290 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.092135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:38.092290 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.092189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:38.092290 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092256 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:46:38.092560 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092304 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:46:38.092560 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092309 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:42.092288809 +0000 UTC m=+43.739852353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : configmap references non-existent config key: service-ca.crt Jul 14 19:46:38.092560 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092334 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls podName:94fdd086-ee53-49c0-af48-d2602acad011 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:42.092324399 +0000 UTC m=+43.739887949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls") pod "dns-default-lb7wt" (UID: "94fdd086-ee53-49c0-af48-d2602acad011") : secret "dns-default-metrics-tls" not found Jul 14 19:46:38.092560 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092361 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:42.092345498 +0000 UTC m=+43.739909053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : secret "router-metrics-certs-default" not found Jul 14 19:46:38.092560 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.092356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:38.092560 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.092395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:38.092560 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092524 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:38.092560 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092552 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 19:46:38.093019 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092583 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls podName:efee90be-3aa3-43a7-9a09-9a537528cff0 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:42.092559737 +0000 UTC m=+43.740123281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-f4mvb" (UID: "efee90be-3aa3-43a7-9a09-9a537528cff0") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:38.093019 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:38.092603 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls podName:1adbaaac-9ef8-44c6-b06d-1927d805a0ee nodeName:}" failed. No retries permitted until 2026-07-14 19:46:42.092593219 +0000 UTC m=+43.740156768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls") pod "cluster-samples-operator-65db858894-mfjjk" (UID: "1adbaaac-9ef8-44c6-b06d-1927d805a0ee") : secret "samples-operator-tls" not found Jul 14 19:46:38.145672 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.145641 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6"] Jul 14 19:46:38.166233 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.166200 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6"] Jul 14 19:46:38.166406 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.166313 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6" Jul 14 19:46:38.168744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.168720 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qktfx\"" Jul 14 19:46:38.193404 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.193374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv5rp\" (UniqueName: \"kubernetes.io/projected/8a94e910-72f4-472e-898e-611a3a8437c1-kube-api-access-cv5rp\") pod \"network-check-source-869dcfcdcd-nt5s6\" (UID: \"8a94e910-72f4-472e-898e-611a3a8437c1\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6" Jul 14 19:46:38.294251 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.294150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv5rp\" (UniqueName: \"kubernetes.io/projected/8a94e910-72f4-472e-898e-611a3a8437c1-kube-api-access-cv5rp\") pod \"network-check-source-869dcfcdcd-nt5s6\" (UID: \"8a94e910-72f4-472e-898e-611a3a8437c1\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6" Jul 14 19:46:38.306295 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.306021 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv5rp\" (UniqueName: \"kubernetes.io/projected/8a94e910-72f4-472e-898e-611a3a8437c1-kube-api-access-cv5rp\") pod \"network-check-source-869dcfcdcd-nt5s6\" (UID: \"8a94e910-72f4-472e-898e-611a3a8437c1\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6" Jul 14 19:46:38.476669 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:38.476629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6" Jul 14 19:46:39.313961 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:39.313929 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:39.406682 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:39.406640 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:41.348341 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:41.348291 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:41.447816 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:41.447780 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:42.027737 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:42.027697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:42.027981 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.027886 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:46:42.027981 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.027905 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdb65b65-wgdv9: secret "image-registry-tls" not found Jul 14 19:46:42.027981 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.027976 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls podName:c77be464-74ee-490f-aed1-99bed17482ef nodeName:}" failed. No retries permitted until 2026-07-14 19:46:50.027958104 +0000 UTC m=+51.675521650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls") pod "image-registry-64cdb65b65-wgdv9" (UID: "c77be464-74ee-490f-aed1-99bed17482ef") : secret "image-registry-tls" not found Jul 14 19:46:42.128679 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:42.128645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:42.128862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:42.128705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:42.128862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:42.128755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:42.128862 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.128808 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:46:42.128862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:42.128813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:42.128862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:42.128857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:42.129120 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.128889 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls podName:94fdd086-ee53-49c0-af48-d2602acad011 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:50.128868918 +0000 UTC m=+51.776432484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls") pod "dns-default-lb7wt" (UID: "94fdd086-ee53-49c0-af48-d2602acad011") : secret "dns-default-metrics-tls" not found Jul 14 19:46:42.129120 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.128910 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:50.128900893 +0000 UTC m=+51.776464438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : configmap references non-existent config key: service-ca.crt Jul 14 19:46:42.129120 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.128927 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 14 19:46:42.129120 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.128944 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:46:42.129120 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.128967 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls podName:1adbaaac-9ef8-44c6-b06d-1927d805a0ee nodeName:}" failed. No retries permitted until 2026-07-14 19:46:50.128958062 +0000 UTC m=+51.776521618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls") pod "cluster-samples-operator-65db858894-mfjjk" (UID: "1adbaaac-9ef8-44c6-b06d-1927d805a0ee") : secret "samples-operator-tls" not found Jul 14 19:46:42.129120 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.128987 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:42.129120 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.128999 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:46:50.128980953 +0000 UTC m=+51.776544501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : secret "router-metrics-certs-default" not found Jul 14 19:46:42.129120 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:42.129095 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls podName:efee90be-3aa3-43a7-9a09-9a537528cff0 nodeName:}" failed. No retries permitted until 2026-07-14 19:46:50.129075833 +0000 UTC m=+51.776639406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-f4mvb" (UID: "efee90be-3aa3-43a7-9a09-9a537528cff0") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:42.674624 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:42.674552 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6"] Jul 14 19:46:42.751046 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:42.750734 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a94e910_72f4_472e_898e_611a3a8437c1.slice/crio-fa58e793dc5d2007e199f69cdfed66600d7df4ee5d6c05d254a58af489cacc85 WatchSource:0}: Error finding container fa58e793dc5d2007e199f69cdfed66600d7df4ee5d6c05d254a58af489cacc85: Status 404 returned error can't find the container with id fa58e793dc5d2007e199f69cdfed66600d7df4ee5d6c05d254a58af489cacc85 Jul 14 19:46:43.105638 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.105582 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/0.log" Jul 14 19:46:43.105638 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.105630 2578 generic.go:356] "Generic (PLEG): container finished" podID="371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97" containerID="4017c44048484d51c7bff48f56bc79c0380ac26d2c7ad1b2b48d44940edcedcd" exitCode=255 Jul 14 19:46:43.105872 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.105723 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-h7mpd" event={"ID":"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97","Type":"ContainerDied","Data":"4017c44048484d51c7bff48f56bc79c0380ac26d2c7ad1b2b48d44940edcedcd"} Jul 14 19:46:43.106276 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.106046 2578 scope.go:117] "RemoveContainer" containerID="4017c44048484d51c7bff48f56bc79c0380ac26d2c7ad1b2b48d44940edcedcd" Jul 14 19:46:43.107486 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.107462 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4" event={"ID":"40c179c1-a98f-437a-874a-ebb88a6b8200","Type":"ContainerStarted","Data":"891b80b9ef81ead8702d4f2f63e9664f60b25a00a2decdc48a68925242504ef3"} Jul 14 19:46:43.109600 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.109575 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lhn87" event={"ID":"3dc21d0b-1485-4386-8f3b-ebdef6134092","Type":"ContainerStarted","Data":"c6a6f3a8e8a69bd6f13aaa99c4a71290a294f5446c569a8f028ebe027424d3a4"} Jul 14 19:46:43.111266 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.111237 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" event={"ID":"4c335c34-95b3-4b24-8b1c-0e503876aeb9","Type":"ContainerStarted","Data":"43b3931d9945ab0cf312aa15660dfccc8456efa5d351b7ca1a995d0b8a504be4"} Jul 14 19:46:43.112610 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.112580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6" event={"ID":"8a94e910-72f4-472e-898e-611a3a8437c1","Type":"ContainerStarted","Data":"fa58e793dc5d2007e199f69cdfed66600d7df4ee5d6c05d254a58af489cacc85"} Jul 14 19:46:43.114011 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.113988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" event={"ID":"85b2851b-b372-4935-9b8b-379379b4ae92","Type":"ContainerStarted","Data":"7e3c875ec0e17eee1e93e78184ce008f399c84bb454a355e9c29ee2a7581bd05"} Jul 14 19:46:43.115122 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.115101 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" event={"ID":"5d04d06d-c61d-4982-895f-50add148475a","Type":"ContainerStarted","Data":"a6587e146d1d09869922223b06dae1d2b7d9e774520a42a2804310b50a93f072"} Jul 14 19:46:43.118323 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.118297 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbce3123-7485-48fa-b9fd-3b3e1e43a9ee" containerID="ca392f3b2499296e3b330616654932861a4b01358fe86fc4e8cce9e174f76083" exitCode=0 Jul 14 19:46:43.118410 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.118334 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerDied","Data":"ca392f3b2499296e3b330616654932861a4b01358fe86fc4e8cce9e174f76083"} Jul 14 19:46:43.151476 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.151455 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-nw56f"] Jul 14 19:46:43.152298 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.152194 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lhn87" podStartSLOduration=1.219961347 podStartE2EDuration="9.1521752s" totalImagesPullingTime="7.932213853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.149244871 +0000 UTC m=+44.796808437" watchObservedRunningTime="2026-07-14 19:46:43.1521752 +0000 UTC m=+44.799738745" Jul 14 19:46:43.200856 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.193892 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:43.200856 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.193759 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-nw56f"] Jul 14 19:46:43.200856 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.200158 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 14 19:46:43.200856 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.200367 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8k6bv\"" Jul 14 19:46:43.202306 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.202252 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" podStartSLOduration=4.296703487 podStartE2EDuration="12.202231877s" totalImagesPullingTime="7.90552839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.201628256 +0000 UTC m=+44.849191823" watchObservedRunningTime="2026-07-14 19:46:43.202231877 +0000 UTC m=+44.849795442" Jul 14 19:46:43.206007 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.205495 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 14 19:46:43.241528 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.241497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07fb4d56-0119-45b4-b2e7-db15a937702c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:43.241975 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.241951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:43.257495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.257444 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" podStartSLOduration=4.294580143 podStartE2EDuration="12.257423941s" totalImagesPullingTime="7.962843798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.230567444 +0000 UTC m=+44.878131009" watchObservedRunningTime="2026-07-14 19:46:43.257423941 +0000 UTC m=+44.904987508" Jul 14 19:46:43.257644 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.257600 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" podStartSLOduration=4.555047042 podStartE2EDuration="12.25759049s" totalImagesPullingTime="7.702543448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.257286686 +0000 UTC m=+44.904850253" watchObservedRunningTime="2026-07-14 19:46:43.25759049 +0000 UTC m=+44.905154057" Jul 14 19:46:43.281931 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.281879 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-j62t4" podStartSLOduration=6.598976726 podStartE2EDuration="12.281856292s" totalImagesPullingTime="5.682879566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:43.281558819 +0000 UTC m=+44.929122388" watchObservedRunningTime="2026-07-14 19:46:43.281856292 +0000 UTC m=+44.929419852" Jul 14 19:46:43.343660 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.343630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07fb4d56-0119-45b4-b2e7-db15a937702c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:43.343918 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.343674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:43.343918 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:43.343908 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 19:46:43.344086 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:43.344009 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert podName:07fb4d56-0119-45b4-b2e7-db15a937702c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:43.843987833 +0000 UTC m=+45.491551381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-nw56f" (UID: "07fb4d56-0119-45b4-b2e7-db15a937702c") : secret "networking-console-plugin-cert" not found Jul 14 19:46:43.345041 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.344984 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07fb4d56-0119-45b4-b2e7-db15a937702c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:43.375600 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.375570 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:43.437979 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.437946 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:43.849305 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:43.849269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:43.849801 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:43.849435 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 19:46:43.849801 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:43.849517 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert podName:07fb4d56-0119-45b4-b2e7-db15a937702c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:44.849495437 +0000 UTC m=+46.497059007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-nw56f" (UID: "07fb4d56-0119-45b4-b2e7-db15a937702c") : secret "networking-console-plugin-cert" not found Jul 14 19:46:44.026048 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.025992 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:44.129022 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.128923 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:44.131469 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.131392 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 19:46:44.132011 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.131990 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/0.log" Jul 14 19:46:44.132116 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.132030 2578 generic.go:356] "Generic (PLEG): container finished" podID="371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97" containerID="22a0e76d0f5433bb8ebc7db5bfec3ed2208068122b316b60e2eb65fdff62479d" exitCode=255 Jul 14 19:46:44.132165 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.132131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-h7mpd" event={"ID":"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97","Type":"ContainerDied","Data":"22a0e76d0f5433bb8ebc7db5bfec3ed2208068122b316b60e2eb65fdff62479d"} Jul 14 19:46:44.132234 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.132221 2578 scope.go:117] "RemoveContainer" containerID="4017c44048484d51c7bff48f56bc79c0380ac26d2c7ad1b2b48d44940edcedcd" Jul 14 19:46:44.132492 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.132472 2578 scope.go:117] "RemoveContainer" containerID="22a0e76d0f5433bb8ebc7db5bfec3ed2208068122b316b60e2eb65fdff62479d" Jul 14 19:46:44.132724 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:44.132690 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-h7mpd_openshift-console-operator(371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97)\"" pod="openshift-console-operator/console-operator-7866497494-h7mpd" podUID="371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97" Jul 14 19:46:44.136621 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.136580 2578 generic.go:356] "Generic (PLEG): container finished" podID="fbce3123-7485-48fa-b9fd-3b3e1e43a9ee" containerID="9e70ec3c74fbe794093defc55ba56bebba443cb8c06b3004eabf3e1a81c7c5a4" exitCode=0 Jul 14 19:46:44.136756 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.136713 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerDied","Data":"9e70ec3c74fbe794093defc55ba56bebba443cb8c06b3004eabf3e1a81c7c5a4"} Jul 14 19:46:44.220333 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.220254 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:44.411869 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.411769 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:44.480789 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.480749 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:44.545082 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.545034 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:44.606098 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.606007 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:44.606098 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.606070 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:44.778565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.778435 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vhxm7"] Jul 14 19:46:44.803954 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.803920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhxm7"] Jul 14 19:46:44.804122 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.804036 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.807436 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.807410 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 19:46:44.807586 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.807470 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bpj6w\"" Jul 14 19:46:44.807586 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.807410 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 19:46:44.830654 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.830621 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:44.859722 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.859693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84af0db3-a613-4183-b841-22d445d83eec-crio-socket\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.860114 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.859761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.860114 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.859779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.860114 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.859955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:44.860114 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.860000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjqm\" (UniqueName: \"kubernetes.io/projected/84af0db3-a613-4183-b841-22d445d83eec-kube-api-access-nhjqm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.860114 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.860032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84af0db3-a613-4183-b841-22d445d83eec-data-volume\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.860114 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:44.860083 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 19:46:44.860378 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:44.860138 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert podName:07fb4d56-0119-45b4-b2e7-db15a937702c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:46.860123701 +0000 UTC m=+48.507687248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-nw56f" (UID: "07fb4d56-0119-45b4-b2e7-db15a937702c") : secret "networking-console-plugin-cert" not found Jul 14 19:46:44.961306 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.961270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjqm\" (UniqueName: \"kubernetes.io/projected/84af0db3-a613-4183-b841-22d445d83eec-kube-api-access-nhjqm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.961495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.961327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84af0db3-a613-4183-b841-22d445d83eec-data-volume\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.961495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.961360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84af0db3-a613-4183-b841-22d445d83eec-crio-socket\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.961495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.961415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.961495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.961437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.961754 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:44.961551 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:46:44.961754 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.961593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84af0db3-a613-4183-b841-22d445d83eec-crio-socket\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.961754 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:44.961621 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:46:45.46160161 +0000 UTC m=+47.109165159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : configmap "kube-rbac-proxy" not found Jul 14 19:46:44.961754 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:44.961696 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:46:44.961992 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:44.961765 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:46:45.461747524 +0000 UTC m=+47.109311068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : secret "insights-runtime-extractor-tls" not found Jul 14 19:46:44.961992 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.961746 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84af0db3-a613-4183-b841-22d445d83eec-data-volume\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:44.963776 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.963698 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:44.971692 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:44.971671 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjqm\" (UniqueName: \"kubernetes.io/projected/84af0db3-a613-4183-b841-22d445d83eec-kube-api-access-nhjqm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:45.143442 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.143416 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 19:46:45.143855 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.143816 2578 scope.go:117] "RemoveContainer" containerID="22a0e76d0f5433bb8ebc7db5bfec3ed2208068122b316b60e2eb65fdff62479d" Jul 14 19:46:45.144078 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:45.144052 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-h7mpd_openshift-console-operator(371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97)\"" pod="openshift-console-operator/console-operator-7866497494-h7mpd" podUID="371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97" Jul 14 19:46:45.147405 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.147372 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" event={"ID":"fbce3123-7485-48fa-b9fd-3b3e1e43a9ee","Type":"ContainerStarted","Data":"01832becbb41b0df5fb65b0feef6af17a9edcf420c2cc46073d20832af2600fa"} Jul 14 19:46:45.194741 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.194674 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gb6f2" podStartSLOduration=2.692544515 podStartE2EDuration="45.194650799s" totalImagesPullingTime="42.502106284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-14 19:46:45.19230893 +0000 UTC m=+46.839872530" watchObservedRunningTime="2026-07-14 19:46:45.194650799 +0000 UTC m=+46.842214365" Jul 14 19:46:45.248304 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.248266 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:45.425901 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.425785 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v"] Jul 14 19:46:45.459030 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.458990 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v"] Jul 14 19:46:45.459262 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.459146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" Jul 14 19:46:45.461798 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.461769 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7s478\"" Jul 14 19:46:45.462210 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.462185 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 14 19:46:45.462340 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.462198 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 14 19:46:45.468396 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.467787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:45.468396 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.467854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:45.468396 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:45.468181 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:46:45.468396 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:45.468251 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:46:46.468233228 +0000 UTC m=+48.115796772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : secret "insights-runtime-extractor-tls" not found Jul 14 19:46:45.468623 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:45.468611 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:46:45.468693 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:45.468675 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:46:46.468654128 +0000 UTC m=+48.116217673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : configmap "kube-rbac-proxy" not found Jul 14 19:46:45.495133 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.489643 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:45.567901 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.567867 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:45.568954 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.568923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmnq\" (UniqueName: \"kubernetes.io/projected/d3850dc3-2e5e-453d-a772-2cbed31dee96-kube-api-access-dmmnq\") pod \"migrator-7f59b54f6c-42h8v\" (UID: \"d3850dc3-2e5e-453d-a772-2cbed31dee96\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" Jul 14 19:46:45.668622 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.668580 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:45.670603 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.670578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmnq\" (UniqueName: \"kubernetes.io/projected/d3850dc3-2e5e-453d-a772-2cbed31dee96-kube-api-access-dmmnq\") pod \"migrator-7f59b54f6c-42h8v\" (UID: \"d3850dc3-2e5e-453d-a772-2cbed31dee96\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" Jul 14 19:46:45.681373 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.681298 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmnq\" (UniqueName: \"kubernetes.io/projected/d3850dc3-2e5e-453d-a772-2cbed31dee96-kube-api-access-dmmnq\") pod \"migrator-7f59b54f6c-42h8v\" (UID: \"d3850dc3-2e5e-453d-a772-2cbed31dee96\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" Jul 14 19:46:45.765238 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.765203 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:45.771868 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:45.771840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" Jul 14 19:46:46.105850 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.105799 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v"] Jul 14 19:46:46.111340 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:46.111255 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3850dc3_2e5e_453d_a772_2cbed31dee96.slice/crio-2d94f2f5001c5c75ee60931a0833bbed25e456b0f7889b11052863a24b6a2451 WatchSource:0}: Error finding container 2d94f2f5001c5c75ee60931a0833bbed25e456b0f7889b11052863a24b6a2451: Status 404 returned error can't find the container with id 2d94f2f5001c5c75ee60931a0833bbed25e456b0f7889b11052863a24b6a2451 Jul 14 19:46:46.151484 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.151449 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" event={"ID":"d3850dc3-2e5e-453d-a772-2cbed31dee96","Type":"ContainerStarted","Data":"2d94f2f5001c5c75ee60931a0833bbed25e456b0f7889b11052863a24b6a2451"} Jul 14 19:46:46.151762 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.151749 2578 scope.go:117] "RemoveContainer" containerID="22a0e76d0f5433bb8ebc7db5bfec3ed2208068122b316b60e2eb65fdff62479d" Jul 14 19:46:46.151952 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:46.151933 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-h7mpd_openshift-console-operator(371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97)\"" pod="openshift-console-operator/console-operator-7866497494-h7mpd" podUID="371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97" Jul 14 19:46:46.181744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.181722 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xthjn_18ce272c-d523-4253-bf80-92b16152b6e5/dns-node-resolver/0.log" Jul 14 19:46:46.433536 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.433506 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:46.457286 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.457260 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-5r8cj"] Jul 14 19:46:46.476907 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.476882 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-5r8cj"] Jul 14 19:46:46.477040 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.477020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.479735 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.479710 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 14 19:46:46.479846 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.479758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:46.479846 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.479777 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 14 19:46:46.479846 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.479796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:46.480018 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:46.479947 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:46:46.480018 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:46.479989 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:46:46.480018 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:46.480006 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:46:48.479985815 +0000 UTC m=+50.127549380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : configmap "kube-rbac-proxy" not found Jul 14 19:46:46.480143 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:46.480039 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:46:48.480023361 +0000 UTC m=+50.127586912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : secret "insights-runtime-extractor-tls" not found Jul 14 19:46:46.480715 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.480694 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-24kn8\"" Jul 14 19:46:46.480813 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.480696 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 14 19:46:46.480813 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.480737 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 14 19:46:46.533085 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.533048 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:46.580820 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.580788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d2d9b958-923a-4922-a28d-e1f0bc3566ae-signing-key\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.581028 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.580863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d2d9b958-923a-4922-a28d-e1f0bc3566ae-signing-cabundle\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.581028 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.580928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2st\" (UniqueName: \"kubernetes.io/projected/d2d9b958-923a-4922-a28d-e1f0bc3566ae-kube-api-access-vs2st\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.682196 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.682104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2st\" (UniqueName: \"kubernetes.io/projected/d2d9b958-923a-4922-a28d-e1f0bc3566ae-kube-api-access-vs2st\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.682399 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.682378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d2d9b958-923a-4922-a28d-e1f0bc3566ae-signing-key\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.682464 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.682422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d2d9b958-923a-4922-a28d-e1f0bc3566ae-signing-cabundle\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.682991 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.682972 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d2d9b958-923a-4922-a28d-e1f0bc3566ae-signing-cabundle\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.684810 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.684784 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d2d9b958-923a-4922-a28d-e1f0bc3566ae-signing-key\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.690820 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.690798 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2st\" (UniqueName: \"kubernetes.io/projected/d2d9b958-923a-4922-a28d-e1f0bc3566ae-kube-api-access-vs2st\") pod \"service-ca-79f68b9f4f-5r8cj\" (UID: \"d2d9b958-923a-4922-a28d-e1f0bc3566ae\") " pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.786577 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.786543 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" Jul 14 19:46:46.884693 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.884649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:46.885363 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:46.884959 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 19:46:46.885363 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:46.885044 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert podName:07fb4d56-0119-45b4-b2e7-db15a937702c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:50.88502146 +0000 UTC m=+52.532585005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-nw56f" (UID: "07fb4d56-0119-45b4-b2e7-db15a937702c") : secret "networking-console-plugin-cert" not found Jul 14 19:46:46.926802 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:46.926770 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-5r8cj"] Jul 14 19:46:46.929865 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:46.929815 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d9b958_923a_4922_a28d_e1f0bc3566ae.slice/crio-9ebd5f9adba365f54bef429bf7de61335cba0f21d58c8e3521f33de765f32522 WatchSource:0}: Error finding container 9ebd5f9adba365f54bef429bf7de61335cba0f21d58c8e3521f33de765f32522: Status 404 returned error can't find the container with id 9ebd5f9adba365f54bef429bf7de61335cba0f21d58c8e3521f33de765f32522 Jul 14 19:46:47.065265 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.065226 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:47.156006 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.155964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6" event={"ID":"8a94e910-72f4-472e-898e-611a3a8437c1","Type":"ContainerStarted","Data":"d67c504c5e77064279d1acd7036c1a9c8c812c1172a7dccf020c4955fbd5b63d"} Jul 14 19:46:47.157636 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.157606 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" event={"ID":"d2d9b958-923a-4922-a28d-e1f0bc3566ae","Type":"ContainerStarted","Data":"d024c443a8f807a14a32ea8556a30540679d233ebd797b95ba913cef99da5a35"} Jul 14 19:46:47.157786 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.157652 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" event={"ID":"d2d9b958-923a-4922-a28d-e1f0bc3566ae","Type":"ContainerStarted","Data":"9ebd5f9adba365f54bef429bf7de61335cba0f21d58c8e3521f33de765f32522"} Jul 14 19:46:47.163795 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.163717 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:47.173462 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.173420 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-nt5s6" podStartSLOduration=5.571216765 podStartE2EDuration="9.173405824s" totalImagesPullingTime="3.602189059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:47.172226601 +0000 UTC m=+48.819790164" watchObservedRunningTime="2026-07-14 19:46:47.173405824 +0000 UTC m=+48.820969389" Jul 14 19:46:47.184473 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.184409 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w2p7r_a52800f2-1350-49d4-ad60-df31b5d00640/node-ca/0.log" Jul 14 19:46:47.192155 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.192111 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-5r8cj" podStartSLOduration=1.192092709 podStartE2EDuration="1.192092709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:47.19046112 +0000 UTC m=+48.838024684" watchObservedRunningTime="2026-07-14 19:46:47.192092709 +0000 UTC m=+48.839656278" Jul 14 19:46:47.452873 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.452762 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:47.512531 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.512468 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:47.589759 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:47.589723 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lhn87_3dc21d0b-1485-4386-8f3b-ebdef6134092/serve-healthcheck-canary/0.log" Jul 14 19:46:48.132266 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:48.132228 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:48.425736 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:48.425589 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:48.500584 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:48.500553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:48.500584 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:48.500586 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:48.500787 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:48.500688 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 19:46:48.500787 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:48.500759 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:46:52.500738784 +0000 UTC m=+54.148302335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : configmap "kube-rbac-proxy" not found Jul 14 19:46:48.500787 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:48.500779 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:46:48.500968 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:48.500845 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:46:52.500811728 +0000 UTC m=+54.148375282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : secret "insights-runtime-extractor-tls" not found Jul 14 19:46:48.534385 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:48.534354 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:48.635267 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:48.635209 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:48.788473 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:48.788425 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:49.165470 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:49.165433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" event={"ID":"d3850dc3-2e5e-453d-a772-2cbed31dee96","Type":"ContainerStarted","Data":"23cf25819da6295ede135b8a6d5d550a6fd8767540ba8908ac1c1128790d32c1"} Jul 14 19:46:49.165651 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:49.165488 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" event={"ID":"d3850dc3-2e5e-453d-a772-2cbed31dee96","Type":"ContainerStarted","Data":"80fab35e9ef5df3a6fef537efa0fa4aade0bfe6c741623574dba6b9453c51fd5"} Jul 14 19:46:49.186454 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:49.186399 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-42h8v" podStartSLOduration=1.819656147 podStartE2EDuration="4.186383393s" totalImagesPullingTime="2.366727246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:49.185888534 +0000 UTC m=+50.833452099" watchObservedRunningTime="2026-07-14 19:46:49.186383393 +0000 UTC m=+50.833946986" Jul 14 19:46:49.607873 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:49.607811 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:49.727307 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:49.727269 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:50.114680 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.114643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:46:50.114878 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.114794 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 19:46:50.114878 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.114811 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64cdb65b65-wgdv9: secret "image-registry-tls" not found Jul 14 19:46:50.114966 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.114893 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls podName:c77be464-74ee-490f-aed1-99bed17482ef nodeName:}" failed. No retries permitted until 2026-07-14 19:47:06.114874609 +0000 UTC m=+67.762438172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls") pod "image-registry-64cdb65b65-wgdv9" (UID: "c77be464-74ee-490f-aed1-99bed17482ef") : secret "image-registry-tls" not found Jul 14 19:46:50.126354 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.126330 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:50.215355 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.215317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:50.215520 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.215396 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:46:50.215520 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.215456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:46:50.215520 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.215497 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:47:06.215478909 +0000 UTC m=+67.863042467 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : configmap references non-existent config key: service-ca.crt Jul 14 19:46:50.215690 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.215548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:50.215690 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.215551 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:50.215690 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.215607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:46:50.215690 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.215555 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 14 19:46:50.215690 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.215671 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 19:46:50.215690 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.215667 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls podName:efee90be-3aa3-43a7-9a09-9a537528cff0 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:06.215646773 +0000 UTC m=+67.863210336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-f4mvb" (UID: "efee90be-3aa3-43a7-9a09-9a537528cff0") : secret "cluster-monitoring-operator-tls" not found Jul 14 19:46:50.215928 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.215707 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls podName:94fdd086-ee53-49c0-af48-d2602acad011 nodeName:}" failed. No retries permitted until 2026-07-14 19:47:06.215697177 +0000 UTC m=+67.863260723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls") pod "dns-default-lb7wt" (UID: "94fdd086-ee53-49c0-af48-d2602acad011") : secret "dns-default-metrics-tls" not found Jul 14 19:46:50.215928 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.215724 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs podName:acafe8a7-3da1-4bac-b4a9-81aaa72fc21f nodeName:}" failed. No retries permitted until 2026-07-14 19:47:06.215715669 +0000 UTC m=+67.863279213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs") pod "router-default-b7cc8dd66-pmcdg" (UID: "acafe8a7-3da1-4bac-b4a9-81aaa72fc21f") : secret "router-metrics-certs-default" not found Jul 14 19:46:50.217958 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.217933 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1adbaaac-9ef8-44c6-b06d-1927d805a0ee-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mfjjk\" (UID: \"1adbaaac-9ef8-44c6-b06d-1927d805a0ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:50.321948 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.321916 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:50.517004 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.516925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" Jul 14 19:46:50.519564 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.519531 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:50.672151 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.672123 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk"] Jul 14 19:46:50.742370 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.742335 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:50.921998 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:50.921953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:50.922180 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.922124 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 19:46:50.922240 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:50.922202 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert podName:07fb4d56-0119-45b4-b2e7-db15a937702c nodeName:}" failed. No retries permitted until 2026-07-14 19:46:58.922182301 +0000 UTC m=+60.569745845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-nw56f" (UID: "07fb4d56-0119-45b4-b2e7-db15a937702c") : secret "networking-console-plugin-cert" not found Jul 14 19:46:51.173773 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:51.173695 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" event={"ID":"1adbaaac-9ef8-44c6-b06d-1927d805a0ee","Type":"ContainerStarted","Data":"c202a644ffefac5e3f17f274e785fdc483a17dbb5d92ae59c9f154cfd4966e1d"} Jul 14 19:46:51.184514 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:51.184479 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:51.737716 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:51.737682 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:51.920718 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:51.920686 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:52.373122 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:52.373061 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:52.476291 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:52.476251 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:52.536524 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:52.536481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:52.536705 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:52.536536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:52.536705 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:52.536689 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 19:46:52.536786 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:46:52.536755 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls podName:84af0db3-a613-4183-b841-22d445d83eec nodeName:}" failed. No retries permitted until 2026-07-14 19:47:00.536738619 +0000 UTC m=+62.184302163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhxm7" (UID: "84af0db3-a613-4183-b841-22d445d83eec") : secret "insights-runtime-extractor-tls" not found Jul 14 19:46:52.536971 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:52.536953 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84af0db3-a613-4183-b841-22d445d83eec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:46:52.573537 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:52.573501 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:52.635892 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:52.635787 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:53.187046 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:53.186960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" event={"ID":"1adbaaac-9ef8-44c6-b06d-1927d805a0ee","Type":"ContainerStarted","Data":"6667dc125e67674a0cfae9f2e173b765f38116e3ec5014fa124d682d83ba69e5"} Jul 14 19:46:53.187046 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:53.187018 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" event={"ID":"1adbaaac-9ef8-44c6-b06d-1927d805a0ee","Type":"ContainerStarted","Data":"ee11d83d89528c6634e3f0d35265bc4a40c589fdb7cc35dca6375b7e2cc44618"} Jul 14 19:46:53.204620 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:53.204570 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mfjjk" podStartSLOduration=20.205412295 podStartE2EDuration="22.204554448s" totalImagesPullingTime="1.999142153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:53.203992581 +0000 UTC m=+54.851556149" watchObservedRunningTime="2026-07-14 19:46:53.204554448 +0000 UTC m=+54.852118021" Jul 14 19:46:53.317024 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:53.316986 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:54.364678 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:54.364644 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:54.465031 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:54.464997 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:55.967586 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:55.967536 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:56.067737 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:56.067709 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:56.189498 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:56.189463 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" limit=1024 Jul 14 19:46:58.879107 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:58.879076 2578 scope.go:117] "RemoveContainer" containerID="22a0e76d0f5433bb8ebc7db5bfec3ed2208068122b316b60e2eb65fdff62479d" Jul 14 19:46:58.994590 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:58.994550 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:58.997088 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:58.997060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/07fb4d56-0119-45b4-b2e7-db15a937702c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-nw56f\" (UID: \"07fb4d56-0119-45b4-b2e7-db15a937702c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:59.121682 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:59.121644 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8k6bv\"" Jul 14 19:46:59.129929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:59.129873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" Jul 14 19:46:59.211488 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:59.211459 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 19:46:59.211645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:59.211580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-h7mpd" event={"ID":"371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97","Type":"ContainerStarted","Data":"5a3071c035bba7d4b3e73e6603030b7b3def8490f1bc05ca674a58363f0f2678"} Jul 14 19:46:59.211886 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:59.211864 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:46:59.233503 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:59.233456 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-h7mpd" podStartSLOduration=20.304379328 podStartE2EDuration="28.233436896s" totalImagesPullingTime="7.929057568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:46:59.232250416 +0000 UTC m=+60.879813982" watchObservedRunningTime="2026-07-14 19:46:59.233436896 +0000 UTC m=+60.881000464" Jul 14 19:46:59.283988 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:59.281780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-nw56f"] Jul 14 19:46:59.287153 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:46:59.287123 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fb4d56_0119_45b4_b2e7_db15a937702c.slice/crio-77f915cfbd3178b0da0227cdb1138269c5aac651300d2b1301b77a932317f3d4 WatchSource:0}: Error finding container 77f915cfbd3178b0da0227cdb1138269c5aac651300d2b1301b77a932317f3d4: Status 404 returned error can't find the container with id 77f915cfbd3178b0da0227cdb1138269c5aac651300d2b1301b77a932317f3d4 Jul 14 19:46:59.515541 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:46:59.515473 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-h7mpd" Jul 14 19:47:00.058029 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:00.058003 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kkjx6" Jul 14 19:47:00.216858 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:00.216807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" event={"ID":"07fb4d56-0119-45b4-b2e7-db15a937702c","Type":"ContainerStarted","Data":"77f915cfbd3178b0da0227cdb1138269c5aac651300d2b1301b77a932317f3d4"} Jul 14 19:47:00.611362 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:00.611326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:47:00.614049 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:00.614020 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84af0db3-a613-4183-b841-22d445d83eec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhxm7\" (UID: \"84af0db3-a613-4183-b841-22d445d83eec\") " pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:47:00.718133 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:00.718103 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bpj6w\"" Jul 14 19:47:00.726276 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:00.726245 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhxm7" Jul 14 19:47:01.077230 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:01.077197 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhxm7"] Jul 14 19:47:01.081081 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:01.081047 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84af0db3_a613_4183_b841_22d445d83eec.slice/crio-6d06e3db80fca370153029c14b5e15b0bf16f1cc34fc47e748f21f83cc0c8e4c WatchSource:0}: Error finding container 6d06e3db80fca370153029c14b5e15b0bf16f1cc34fc47e748f21f83cc0c8e4c: Status 404 returned error can't find the container with id 6d06e3db80fca370153029c14b5e15b0bf16f1cc34fc47e748f21f83cc0c8e4c Jul 14 19:47:01.222363 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:01.222266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" event={"ID":"07fb4d56-0119-45b4-b2e7-db15a937702c","Type":"ContainerStarted","Data":"417212e2e7c5324a1ce40867d3101b4b92f0f5bf27097e7cbd55a1d77c0d8b55"} Jul 14 19:47:01.223382 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:01.223354 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhxm7" event={"ID":"84af0db3-a613-4183-b841-22d445d83eec","Type":"ContainerStarted","Data":"6d06e3db80fca370153029c14b5e15b0bf16f1cc34fc47e748f21f83cc0c8e4c"} Jul 14 19:47:01.239781 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:01.239739 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-nw56f" podStartSLOduration=16.553202011 podStartE2EDuration="18.239725512s" totalImagesPullingTime="1.686523501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:01.238997692 +0000 UTC m=+62.886561258" watchObservedRunningTime="2026-07-14 19:47:01.239725512 +0000 UTC m=+62.887289088" Jul 14 19:47:03.641787 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.641735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:47:03.642271 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.641818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:47:03.644787 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.644756 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 19:47:03.644922 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.644841 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 19:47:03.655095 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.655061 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff66e5b-b8e1-43f9-bdba-b82de3668ef3-metrics-certs\") pod \"network-metrics-daemon-mktvj\" (UID: \"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3\") " pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:47:03.655271 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.655120 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/060c531f-952a-432e-a4a3-c08cd392bdcd-original-pull-secret\") pod \"global-pull-secret-syncer-qzk58\" (UID: \"060c531f-952a-432e-a4a3-c08cd392bdcd\") " pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:47:03.742502 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.742462 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:47:03.745435 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.745407 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2f2m\" (UniqueName: \"kubernetes.io/projected/c0cf5e05-c1b6-4453-aea9-7f5629f80f4c-kube-api-access-w2f2m\") pod \"network-check-target-f4clt\" (UID: \"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c\") " pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:47:03.796511 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.796465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qzk58" Jul 14 19:47:03.808487 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.808441 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xm9xl\"" Jul 14 19:47:03.816627 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.816599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:47:03.816858 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.816817 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mvlkh\"" Jul 14 19:47:03.825202 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:03.825180 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mktvj" Jul 14 19:47:04.999045 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:04.998922 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qzk58"] Jul 14 19:47:05.001714 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:05.001691 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060c531f_952a_432e_a4a3_c08cd392bdcd.slice/crio-58e0048de71034085816e08ec8bbf71248349f40725fbef62f2496fcfb801a01 WatchSource:0}: Error finding container 58e0048de71034085816e08ec8bbf71248349f40725fbef62f2496fcfb801a01: Status 404 returned error can't find the container with id 58e0048de71034085816e08ec8bbf71248349f40725fbef62f2496fcfb801a01 Jul 14 19:47:05.009461 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.009429 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mktvj"] Jul 14 19:47:05.012245 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:05.012212 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff66e5b_b8e1_43f9_bdba_b82de3668ef3.slice/crio-0e4cf9e827d7bed0b90ab696fb3b2ad0f53c57a2cb06eb384adedfbf88f8b85f WatchSource:0}: Error finding container 0e4cf9e827d7bed0b90ab696fb3b2ad0f53c57a2cb06eb384adedfbf88f8b85f: Status 404 returned error can't find the container with id 0e4cf9e827d7bed0b90ab696fb3b2ad0f53c57a2cb06eb384adedfbf88f8b85f Jul 14 19:47:05.030102 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.029641 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f4clt"] Jul 14 19:47:05.031771 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:05.031744 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cf5e05_c1b6_4453_aea9_7f5629f80f4c.slice/crio-5ad0a2c0281d4ce3dd42a807d4479d4f9cf6ebeba370f19c6ed4c669ae18457c WatchSource:0}: Error finding container 5ad0a2c0281d4ce3dd42a807d4479d4f9cf6ebeba370f19c6ed4c669ae18457c: Status 404 returned error can't find the container with id 5ad0a2c0281d4ce3dd42a807d4479d4f9cf6ebeba370f19c6ed4c669ae18457c Jul 14 19:47:05.240821 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.240779 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhxm7" event={"ID":"84af0db3-a613-4183-b841-22d445d83eec","Type":"ContainerStarted","Data":"ed7bedd7c35af40dc264d6dc499e85da4cd40d957afd1b220f556d2f685cfe70"} Jul 14 19:47:05.241994 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.241971 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qzk58" event={"ID":"060c531f-952a-432e-a4a3-c08cd392bdcd","Type":"ContainerStarted","Data":"58e0048de71034085816e08ec8bbf71248349f40725fbef62f2496fcfb801a01"} Jul 14 19:47:05.243421 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.243400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f4clt" event={"ID":"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c","Type":"ContainerStarted","Data":"db2bce7d6813be5b2b2dafd699e8e496baf07d165a8e64d82c3889b3981720c8"} Jul 14 19:47:05.243542 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.243434 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f4clt" event={"ID":"c0cf5e05-c1b6-4453-aea9-7f5629f80f4c","Type":"ContainerStarted","Data":"5ad0a2c0281d4ce3dd42a807d4479d4f9cf6ebeba370f19c6ed4c669ae18457c"} Jul 14 19:47:05.243542 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.243501 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:47:05.244562 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.244542 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mktvj" event={"ID":"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3","Type":"ContainerStarted","Data":"0e4cf9e827d7bed0b90ab696fb3b2ad0f53c57a2cb06eb384adedfbf88f8b85f"} Jul 14 19:47:05.260800 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:05.260755 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f4clt" podStartSLOduration=65.260739559 podStartE2EDuration="1m5.260739559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:05.260338827 +0000 UTC m=+66.907902394" watchObservedRunningTime="2026-07-14 19:47:05.260739559 +0000 UTC m=+66.908303184" Jul 14 19:47:06.165728 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.165703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:47:06.168102 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.168081 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"image-registry-64cdb65b65-wgdv9\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:47:06.249784 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.249743 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mktvj" event={"ID":"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3","Type":"ContainerStarted","Data":"513f74c2ab4a938f5b55fde468ee36561b21316a2035f5a62e87ecdd4b25524c"} Jul 14 19:47:06.249982 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.249802 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mktvj" event={"ID":"8ff66e5b-b8e1-43f9-bdba-b82de3668ef3","Type":"ContainerStarted","Data":"9e52c2fdb9bd5a6d7a83793818ae6b08d2bb1b0479fb7f2483665018d34fa0a4"} Jul 14 19:47:06.266163 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.266077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:47:06.266163 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.266131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:47:06.266377 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.266187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:06.266377 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.266233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:06.267137 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.267108 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-service-ca-bundle\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:06.268551 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.268515 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/94fdd086-ee53-49c0-af48-d2602acad011-metrics-tls\") pod \"dns-default-lb7wt\" (UID: \"94fdd086-ee53-49c0-af48-d2602acad011\") " pod="openshift-dns/dns-default-lb7wt" Jul 14 19:47:06.268655 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.268604 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/efee90be-3aa3-43a7-9a09-9a537528cff0-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-f4mvb\" (UID: \"efee90be-3aa3-43a7-9a09-9a537528cff0\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:47:06.268700 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.268681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acafe8a7-3da1-4bac-b4a9-81aaa72fc21f-metrics-certs\") pod \"router-default-b7cc8dd66-pmcdg\" (UID: \"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f\") " pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:06.298673 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.298622 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mktvj" podStartSLOduration=65.309497699 podStartE2EDuration="1m6.298603149s" totalImagesPullingTime="989.10545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:06.297204815 +0000 UTC m=+67.944768381" watchObservedRunningTime="2026-07-14 19:47:06.298603149 +0000 UTC m=+67.946166719" Jul 14 19:47:06.361687 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.361653 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4vjs\"" Jul 14 19:47:06.368996 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.368961 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:47:06.386121 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.386097 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7w2mk\"" Jul 14 19:47:06.388416 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.388389 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n9mmf\"" Jul 14 19:47:06.390667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.390645 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lb7wt" Jul 14 19:47:06.396894 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.396875 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" Jul 14 19:47:06.442861 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.442042 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hfm59\"" Jul 14 19:47:06.449589 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.449554 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:06.461404 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.460471 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg"] Jul 14 19:47:06.469014 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.464063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" Jul 14 19:47:06.469014 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.464134 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k"] Jul 14 19:47:06.469014 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.467959 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 19:47:06.469014 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.468098 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-dcqfs\"" Jul 14 19:47:06.469014 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.468172 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 19:47:06.469014 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.468272 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 14 19:47:06.469014 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.468613 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 19:47:06.471275 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.470502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.481944 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.481397 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 14 19:47:06.481944 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.481429 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 14 19:47:06.481944 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.481656 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 14 19:47:06.482156 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.482132 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 14 19:47:06.485936 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.485882 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7"] Jul 14 19:47:06.493672 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.492360 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.493965 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.493912 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg"] Jul 14 19:47:06.495634 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.495609 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k"] Jul 14 19:47:06.495968 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.495766 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 14 19:47:06.497550 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.497495 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7"] Jul 14 19:47:06.564702 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.564671 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-t88pd"] Jul 14 19:47:06.568141 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.568115 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-t88pd" Jul 14 19:47:06.569705 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.569621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vln\" (UniqueName: \"kubernetes.io/projected/742d7a4b-ff08-46a2-af6c-c40c38c3592b-kube-api-access-n6vln\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.570148 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.569950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.570148 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.569997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-ca\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.570148 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.570032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cd10463-41a7-4db2-a38d-9c0a74f6625d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dd478795d-qfdvg\" (UID: \"4cd10463-41a7-4db2-a38d-9c0a74f6625d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" Jul 14 19:47:06.570148 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.570058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8x4x\" (UniqueName: \"kubernetes.io/projected/4cd10463-41a7-4db2-a38d-9c0a74f6625d-kube-api-access-b8x4x\") pod \"managed-serviceaccount-addon-agent-6dd478795d-qfdvg\" (UID: \"4cd10463-41a7-4db2-a38d-9c0a74f6625d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" Jul 14 19:47:06.570148 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.570120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-hub\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.570457 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.570179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-tmp\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.570457 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.570223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.570457 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.570269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/742d7a4b-ff08-46a2-af6c-c40c38c3592b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.570457 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.570300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.570457 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.570326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fks4m\" (UniqueName: \"kubernetes.io/projected/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-kube-api-access-fks4m\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.571414 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.571208 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c76vc\"" Jul 14 19:47:06.571899 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.571687 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 19:47:06.572282 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.572090 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 19:47:06.581904 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.581811 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-t88pd"] Jul 14 19:47:06.602521 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.602362 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb"] Jul 14 19:47:06.610253 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:06.610223 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefee90be_3aa3_43a7_9a09_9a537528cff0.slice/crio-f27f4d199bd7ec9559c713c03b23d51236480c47e74f3dc279f9263cbefccf64 WatchSource:0}: Error finding container f27f4d199bd7ec9559c713c03b23d51236480c47e74f3dc279f9263cbefccf64: Status 404 returned error can't find the container with id f27f4d199bd7ec9559c713c03b23d51236480c47e74f3dc279f9263cbefccf64 Jul 14 19:47:06.671332 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vln\" (UniqueName: \"kubernetes.io/projected/742d7a4b-ff08-46a2-af6c-c40c38c3592b-kube-api-access-n6vln\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.671332 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.671332 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-ca\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.671332 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671257 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnrlv\" (UniqueName: \"kubernetes.io/projected/1e79965c-8b4b-4093-b6d7-e9789c8dbb8e-kube-api-access-fnrlv\") pod \"downloads-68c9cbb497-t88pd\" (UID: \"1e79965c-8b4b-4093-b6d7-e9789c8dbb8e\") " pod="openshift-console/downloads-68c9cbb497-t88pd" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cd10463-41a7-4db2-a38d-9c0a74f6625d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dd478795d-qfdvg\" (UID: \"4cd10463-41a7-4db2-a38d-9c0a74f6625d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x4x\" (UniqueName: \"kubernetes.io/projected/4cd10463-41a7-4db2-a38d-9c0a74f6625d-kube-api-access-b8x4x\") pod \"managed-serviceaccount-addon-agent-6dd478795d-qfdvg\" (UID: \"4cd10463-41a7-4db2-a38d-9c0a74f6625d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-hub\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-tmp\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/742d7a4b-ff08-46a2-af6c-c40c38c3592b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.671915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fks4m\" (UniqueName: \"kubernetes.io/projected/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-kube-api-access-fks4m\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.672767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.672714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/742d7a4b-ff08-46a2-af6c-c40c38c3592b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.673269 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.672934 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-tmp\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.676031 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.675988 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.676031 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.675975 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.676281 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.676244 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cd10463-41a7-4db2-a38d-9c0a74f6625d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6dd478795d-qfdvg\" (UID: \"4cd10463-41a7-4db2-a38d-9c0a74f6625d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" Jul 14 19:47:06.677642 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.677623 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-klusterlet-config\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.677908 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.677884 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-hub\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.678425 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.678404 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/742d7a4b-ff08-46a2-af6c-c40c38c3592b-ca\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.691717 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.691681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fks4m\" (UniqueName: \"kubernetes.io/projected/2b93186e-77a4-4fe6-b9fb-fce2430ecf69-kube-api-access-fks4m\") pod \"klusterlet-addon-workmgr-56c98696b8-2pmr7\" (UID: \"2b93186e-77a4-4fe6-b9fb-fce2430ecf69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.692543 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.692520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vln\" (UniqueName: \"kubernetes.io/projected/742d7a4b-ff08-46a2-af6c-c40c38c3592b-kube-api-access-n6vln\") pod \"cluster-proxy-proxy-agent-bd5fffd86-2tt8k\" (UID: \"742d7a4b-ff08-46a2-af6c-c40c38c3592b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.692998 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.692951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8x4x\" (UniqueName: \"kubernetes.io/projected/4cd10463-41a7-4db2-a38d-9c0a74f6625d-kube-api-access-b8x4x\") pod \"managed-serviceaccount-addon-agent-6dd478795d-qfdvg\" (UID: \"4cd10463-41a7-4db2-a38d-9c0a74f6625d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" Jul 14 19:47:06.772527 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.772411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnrlv\" (UniqueName: \"kubernetes.io/projected/1e79965c-8b4b-4093-b6d7-e9789c8dbb8e-kube-api-access-fnrlv\") pod \"downloads-68c9cbb497-t88pd\" (UID: \"1e79965c-8b4b-4093-b6d7-e9789c8dbb8e\") " pod="openshift-console/downloads-68c9cbb497-t88pd" Jul 14 19:47:06.774232 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.774195 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b7cc8dd66-pmcdg"] Jul 14 19:47:06.778596 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:06.778558 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacafe8a7_3da1_4bac_b4a9_81aaa72fc21f.slice/crio-66653b728b3bf129b5c07f0578faa682d353b172e4594c2bc91d5a55603ac510 WatchSource:0}: Error finding container 66653b728b3bf129b5c07f0578faa682d353b172e4594c2bc91d5a55603ac510: Status 404 returned error can't find the container with id 66653b728b3bf129b5c07f0578faa682d353b172e4594c2bc91d5a55603ac510 Jul 14 19:47:06.780331 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.780253 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64cdb65b65-wgdv9"] Jul 14 19:47:06.782742 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:06.782713 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77be464_74ee_490f_aed1_99bed17482ef.slice/crio-3b1250755a742f42674068c13fc3a1ff60c4840892dfe29e06d200ecfa805c67 WatchSource:0}: Error finding container 3b1250755a742f42674068c13fc3a1ff60c4840892dfe29e06d200ecfa805c67: Status 404 returned error can't find the container with id 3b1250755a742f42674068c13fc3a1ff60c4840892dfe29e06d200ecfa805c67 Jul 14 19:47:06.786621 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.786584 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnrlv\" (UniqueName: \"kubernetes.io/projected/1e79965c-8b4b-4093-b6d7-e9789c8dbb8e-kube-api-access-fnrlv\") pod \"downloads-68c9cbb497-t88pd\" (UID: \"1e79965c-8b4b-4093-b6d7-e9789c8dbb8e\") " pod="openshift-console/downloads-68c9cbb497-t88pd" Jul 14 19:47:06.810416 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.810341 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lb7wt"] Jul 14 19:47:06.810523 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.810426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" Jul 14 19:47:06.812993 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:06.812968 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fdd086_ee53_49c0_af48_d2602acad011.slice/crio-17c64fac38b33a4f93443d56c726bc49952ccb21b4960cebff74079a44b792f3 WatchSource:0}: Error finding container 17c64fac38b33a4f93443d56c726bc49952ccb21b4960cebff74079a44b792f3: Status 404 returned error can't find the container with id 17c64fac38b33a4f93443d56c726bc49952ccb21b4960cebff74079a44b792f3 Jul 14 19:47:06.820043 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.820018 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" Jul 14 19:47:06.837976 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.837859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:06.884591 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:06.884361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-t88pd" Jul 14 19:47:07.042121 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.042064 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k"] Jul 14 19:47:07.045774 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:07.045737 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod742d7a4b_ff08_46a2_af6c_c40c38c3592b.slice/crio-55a2d94151b7c582b43a5372c08a674a4e14a53acd327e18f6a5343c0a268612 WatchSource:0}: Error finding container 55a2d94151b7c582b43a5372c08a674a4e14a53acd327e18f6a5343c0a268612: Status 404 returned error can't find the container with id 55a2d94151b7c582b43a5372c08a674a4e14a53acd327e18f6a5343c0a268612 Jul 14 19:47:07.070214 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.070145 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg"] Jul 14 19:47:07.077699 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:07.077666 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cd10463_41a7_4db2_a38d_9c0a74f6625d.slice/crio-5cc4524386f03772c559d029c9e1029d55de2c4b225729ccd182d0294e85d360 WatchSource:0}: Error finding container 5cc4524386f03772c559d029c9e1029d55de2c4b225729ccd182d0294e85d360: Status 404 returned error can't find the container with id 5cc4524386f03772c559d029c9e1029d55de2c4b225729ccd182d0294e85d360 Jul 14 19:47:07.090126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.090098 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7"] Jul 14 19:47:07.092896 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:07.092867 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b93186e_77a4_4fe6_b9fb_fce2430ecf69.slice/crio-d7b756fbe8137fe3fb02e2a790b668f387bf7949edb9851b58d62d84905ac9df WatchSource:0}: Error finding container d7b756fbe8137fe3fb02e2a790b668f387bf7949edb9851b58d62d84905ac9df: Status 404 returned error can't find the container with id d7b756fbe8137fe3fb02e2a790b668f387bf7949edb9851b58d62d84905ac9df Jul 14 19:47:07.126203 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.124549 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-t88pd"] Jul 14 19:47:07.129806 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:07.129777 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e79965c_8b4b_4093_b6d7_e9789c8dbb8e.slice/crio-2cdfae73fe793f3a27036cc52ac77ef23dd4a36e045b894e0e1b02fa23718339 WatchSource:0}: Error finding container 2cdfae73fe793f3a27036cc52ac77ef23dd4a36e045b894e0e1b02fa23718339: Status 404 returned error can't find the container with id 2cdfae73fe793f3a27036cc52ac77ef23dd4a36e045b894e0e1b02fa23718339 Jul 14 19:47:07.256754 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.256711 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" event={"ID":"efee90be-3aa3-43a7-9a09-9a537528cff0","Type":"ContainerStarted","Data":"f27f4d199bd7ec9559c713c03b23d51236480c47e74f3dc279f9263cbefccf64"} Jul 14 19:47:07.258285 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.258165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lb7wt" event={"ID":"94fdd086-ee53-49c0-af48-d2602acad011","Type":"ContainerStarted","Data":"17c64fac38b33a4f93443d56c726bc49952ccb21b4960cebff74079a44b792f3"} Jul 14 19:47:07.260627 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.260498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" event={"ID":"c77be464-74ee-490f-aed1-99bed17482ef","Type":"ContainerStarted","Data":"93d1196253104bc645a59d55a085637aad4bc15c82a59a2e4444cde85b5f695e"} Jul 14 19:47:07.260627 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.260550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" event={"ID":"c77be464-74ee-490f-aed1-99bed17482ef","Type":"ContainerStarted","Data":"3b1250755a742f42674068c13fc3a1ff60c4840892dfe29e06d200ecfa805c67"} Jul 14 19:47:07.260627 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.260600 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:47:07.262873 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.262820 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" event={"ID":"742d7a4b-ff08-46a2-af6c-c40c38c3592b","Type":"ContainerStarted","Data":"55a2d94151b7c582b43a5372c08a674a4e14a53acd327e18f6a5343c0a268612"} Jul 14 19:47:07.264761 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.264411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" event={"ID":"2b93186e-77a4-4fe6-b9fb-fce2430ecf69","Type":"ContainerStarted","Data":"d7b756fbe8137fe3fb02e2a790b668f387bf7949edb9851b58d62d84905ac9df"} Jul 14 19:47:07.271442 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.271409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" event={"ID":"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f","Type":"ContainerStarted","Data":"537804da51b04ab0bd830564164b59099166871124f0f129c54014cf417f0c32"} Jul 14 19:47:07.271563 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.271464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" event={"ID":"acafe8a7-3da1-4bac-b4a9-81aaa72fc21f","Type":"ContainerStarted","Data":"66653b728b3bf129b5c07f0578faa682d353b172e4594c2bc91d5a55603ac510"} Jul 14 19:47:07.275103 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.275030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhxm7" event={"ID":"84af0db3-a613-4183-b841-22d445d83eec","Type":"ContainerStarted","Data":"784f87a8fce4858f26bd23c339ee9a23e7052beba587f0982e0ccd402cf48495"} Jul 14 19:47:07.277199 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.277140 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-t88pd" event={"ID":"1e79965c-8b4b-4093-b6d7-e9789c8dbb8e","Type":"ContainerStarted","Data":"2cdfae73fe793f3a27036cc52ac77ef23dd4a36e045b894e0e1b02fa23718339"} Jul 14 19:47:07.278892 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.278866 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" event={"ID":"4cd10463-41a7-4db2-a38d-9c0a74f6625d","Type":"ContainerStarted","Data":"5cc4524386f03772c559d029c9e1029d55de2c4b225729ccd182d0294e85d360"} Jul 14 19:47:07.285938 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.285185 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" podStartSLOduration=67.285166508 podStartE2EDuration="1m7.285166508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:07.283974088 +0000 UTC m=+68.931537667" watchObservedRunningTime="2026-07-14 19:47:07.285166508 +0000 UTC m=+68.932730075" Jul 14 19:47:07.308569 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.308408 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" podStartSLOduration=36.308389981 podStartE2EDuration="36.308389981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:07.306614027 +0000 UTC m=+68.954177594" watchObservedRunningTime="2026-07-14 19:47:07.308389981 +0000 UTC m=+68.955953527" Jul 14 19:47:07.450763 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.450726 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:07.453784 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:07.453550 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:08.289056 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:08.288744 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:08.290154 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:08.289958 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b7cc8dd66-pmcdg" Jul 14 19:47:26.373997 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:26.373936 2578 patch_prober.go:28] interesting pod/image-registry-64cdb65b65-wgdv9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 19:47:26.374454 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:26.374000 2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" podUID="c77be464-74ee-490f-aed1-99bed17482ef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:47:28.294779 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.294740 2578 patch_prober.go:28] interesting pod/image-registry-64cdb65b65-wgdv9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 19:47:28.295336 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.294803 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" podUID="c77be464-74ee-490f-aed1-99bed17482ef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:47:28.398978 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.398899 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qzk58" event={"ID":"060c531f-952a-432e-a4a3-c08cd392bdcd","Type":"ContainerStarted","Data":"f8b8398390ce49e6c0f3fe85aedf30fa7ee763fb433d71b52881fe9c8d1f661b"} Jul 14 19:47:28.401234 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.401175 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" event={"ID":"efee90be-3aa3-43a7-9a09-9a537528cff0","Type":"ContainerStarted","Data":"79aa8d6ce50e5fb0feb262be6582f82e2c9575fc67401442a6f6375d03a3f0d5"} Jul 14 19:47:28.403106 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.403067 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lb7wt" event={"ID":"94fdd086-ee53-49c0-af48-d2602acad011","Type":"ContainerStarted","Data":"81f900e6caf64011691ac095e956048e2cda4db774015e87d1fac96ccb234300"} Jul 14 19:47:28.403106 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.403108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lb7wt" event={"ID":"94fdd086-ee53-49c0-af48-d2602acad011","Type":"ContainerStarted","Data":"e6aad39e287c4f5600b823300d54ce5ecc0ecfc61dee96ba9f85f7fff008c4ac"} Jul 14 19:47:28.403580 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.403505 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lb7wt" Jul 14 19:47:28.405785 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.405760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" event={"ID":"742d7a4b-ff08-46a2-af6c-c40c38c3592b","Type":"ContainerStarted","Data":"dad8d47f51d3d561cdd9b7de246a4d662c5108a02b3728afddf76fbbb6a5a219"} Jul 14 19:47:28.405951 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.405932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" event={"ID":"742d7a4b-ff08-46a2-af6c-c40c38c3592b","Type":"ContainerStarted","Data":"7a84cc12e14500d9fe5e6e4e595e143d7a0fd5000be7b352d02a25f0d7adb1f1"} Jul 14 19:47:28.406030 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.405963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" event={"ID":"742d7a4b-ff08-46a2-af6c-c40c38c3592b","Type":"ContainerStarted","Data":"6f9393cba593c1adb130650dd57596a0e36f13bd43f1f60ae264581437becdb6"} Jul 14 19:47:28.407862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.407614 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" event={"ID":"2b93186e-77a4-4fe6-b9fb-fce2430ecf69","Type":"ContainerStarted","Data":"8f50358852aa67c2ef52ea545f8ec3742aa838600d671c76f6c463d210cb74cf"} Jul 14 19:47:28.407862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.407786 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:28.409378 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.409326 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" Jul 14 19:47:28.409782 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.409761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhxm7" event={"ID":"84af0db3-a613-4183-b841-22d445d83eec","Type":"ContainerStarted","Data":"e96ef5dfdcf0807f70cc0bdf70ae4b9488361d962573ccc98105e5fccf86ef67"} Jul 14 19:47:28.411601 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.411283 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-t88pd" event={"ID":"1e79965c-8b4b-4093-b6d7-e9789c8dbb8e","Type":"ContainerStarted","Data":"05d2fe51672eac82f06f4a1fc8e643a9e5e683b30daebb2be0bd2de0fe95471c"} Jul 14 19:47:28.411601 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.411429 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-t88pd" Jul 14 19:47:28.412974 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.412885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" event={"ID":"4cd10463-41a7-4db2-a38d-9c0a74f6625d","Type":"ContainerStarted","Data":"d286795ecb3df3d032380083d22a8e38533e8dc529b4ceb84c99feb0f45d83b5"} Jul 14 19:47:28.418316 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.418267 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qzk58" podStartSLOduration=65.818527608 podStartE2EDuration="1m28.418250227s" totalImagesPullingTime="22.599722619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:28.417332153 +0000 UTC m=+90.064895774" watchObservedRunningTime="2026-07-14 19:47:28.418250227 +0000 UTC m=+90.065813795" Jul 14 19:47:28.423802 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.423782 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-t88pd" Jul 14 19:47:28.439108 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.439036 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-bd5fffd86-2tt8k" podStartSLOduration=1.88188997 podStartE2EDuration="22.43901539s" totalImagesPullingTime="20.55712542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:28.438302017 +0000 UTC m=+90.085865616" watchObservedRunningTime="2026-07-14 19:47:28.43901539 +0000 UTC m=+90.086578956" Jul 14 19:47:28.478369 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.478308 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56c98696b8-2pmr7" podStartSLOduration=1.95430826 podStartE2EDuration="22.478287636s" totalImagesPullingTime="20.523979376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:28.457587043 +0000 UTC m=+90.105150610" watchObservedRunningTime="2026-07-14 19:47:28.478287636 +0000 UTC m=+90.125851202" Jul 14 19:47:28.500030 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.499970 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lb7wt" podStartSLOduration=33.711232515 podStartE2EDuration="54.499953919s" totalImagesPullingTime="20.788721404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:28.479143203 +0000 UTC m=+90.126706769" watchObservedRunningTime="2026-07-14 19:47:28.499953919 +0000 UTC m=+90.147517484" Jul 14 19:47:28.501104 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.501059 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-f4mvb" podStartSLOduration=36.547405688 podStartE2EDuration="57.501043921s" totalImagesPullingTime="20.953638233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:28.499165923 +0000 UTC m=+90.146729490" watchObservedRunningTime="2026-07-14 19:47:28.501043921 +0000 UTC m=+90.148607489" Jul 14 19:47:28.542089 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.541997 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vhxm7" podStartSLOduration=18.22638576 podStartE2EDuration="44.541972151s" totalImagesPullingTime="26.315586391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:46:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:28.52252669 +0000 UTC m=+90.170090255" watchObservedRunningTime="2026-07-14 19:47:28.541972151 +0000 UTC m=+90.189535719" Jul 14 19:47:28.542307 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.542203 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-t88pd" podStartSLOduration=1.7210999 podStartE2EDuration="22.542189218s" totalImagesPullingTime="20.821089318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:28.541068236 +0000 UTC m=+90.188632034" watchObservedRunningTime="2026-07-14 19:47:28.542189218 +0000 UTC m=+90.189752776" Jul 14 19:47:28.557671 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:28.557602 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6dd478795d-qfdvg" podStartSLOduration=2.033917682 podStartE2EDuration="22.557581479s" totalImagesPullingTime="20.523663797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:28.556449203 +0000 UTC m=+90.204012780" watchObservedRunningTime="2026-07-14 19:47:28.557581479 +0000 UTC m=+90.205145046" Jul 14 19:47:35.303005 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:35.302967 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64cdb65b65-wgdv9"] Jul 14 19:47:35.309130 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:35.309094 2578 patch_prober.go:28] interesting pod/image-registry-64cdb65b65-wgdv9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 19:47:35.309374 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:35.309337 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" podUID="c77be464-74ee-490f-aed1-99bed17482ef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:47:36.252452 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:36.252416 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f4clt" Jul 14 19:47:39.425798 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:39.425767 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lb7wt" Jul 14 19:47:42.843139 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:42.843099 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tqq6d"] Jul 14 19:47:42.919487 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:42.919449 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:42.922328 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:42.922299 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 19:47:42.922474 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:42.922337 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 19:47:42.922711 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:42.922693 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6x7x8\"" Jul 14 19:47:42.923861 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:42.923819 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 19:47:42.923957 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:42.923920 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 19:47:43.012971 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.012928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-wtmp\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.012971 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.012975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-sys\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.013243 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.013006 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-textfile\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.013243 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.013073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-tls\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.013243 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.013137 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.013243 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.013167 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qklm\" (UniqueName: \"kubernetes.io/projected/595ebd05-e643-4645-9025-b3bddb2e02db-kube-api-access-5qklm\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.013243 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.013203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.013403 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.013280 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/595ebd05-e643-4645-9025-b3bddb2e02db-metrics-client-ca\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.013403 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.013310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-root\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114294 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/595ebd05-e643-4645-9025-b3bddb2e02db-metrics-client-ca\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114294 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-root\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114294 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-wtmp\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114307 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-sys\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-textfile\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-tls\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114344 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-root\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114409 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-sys\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qklm\" (UniqueName: \"kubernetes.io/projected/595ebd05-e643-4645-9025-b3bddb2e02db-kube-api-access-5qklm\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.114649 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.115024 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-textfile\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.115024 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.114705 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-wtmp\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.115315 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.115291 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/595ebd05-e643-4645-9025-b3bddb2e02db-metrics-client-ca\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.115400 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.115324 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.116999 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.116969 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.117245 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.117216 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/595ebd05-e643-4645-9025-b3bddb2e02db-node-exporter-tls\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.125115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.125086 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qklm\" (UniqueName: \"kubernetes.io/projected/595ebd05-e643-4645-9025-b3bddb2e02db-kube-api-access-5qklm\") pod \"node-exporter-tqq6d\" (UID: \"595ebd05-e643-4645-9025-b3bddb2e02db\") " pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.234108 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.234070 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqq6d" Jul 14 19:47:43.273138 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:47:43.273095 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595ebd05_e643_4645_9025_b3bddb2e02db.slice/crio-294e10e427061a75885d2739d40a89dd874bfe20ef3aeda4d733b93423ba69c4 WatchSource:0}: Error finding container 294e10e427061a75885d2739d40a89dd874bfe20ef3aeda4d733b93423ba69c4: Status 404 returned error can't find the container with id 294e10e427061a75885d2739d40a89dd874bfe20ef3aeda4d733b93423ba69c4 Jul 14 19:47:43.485684 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:43.485599 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqq6d" event={"ID":"595ebd05-e643-4645-9025-b3bddb2e02db","Type":"ContainerStarted","Data":"294e10e427061a75885d2739d40a89dd874bfe20ef3aeda4d733b93423ba69c4"} Jul 14 19:47:45.307725 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:45.307691 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:47:45.499224 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:45.499193 2578 generic.go:356] "Generic (PLEG): container finished" podID="595ebd05-e643-4645-9025-b3bddb2e02db" containerID="5f4c218e407084b62213c32dd53905c7af6e4c01e7214b4b92bb7a8b48b98062" exitCode=0 Jul 14 19:47:45.499388 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:45.499250 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqq6d" event={"ID":"595ebd05-e643-4645-9025-b3bddb2e02db","Type":"ContainerDied","Data":"5f4c218e407084b62213c32dd53905c7af6e4c01e7214b4b92bb7a8b48b98062"} Jul 14 19:47:46.505718 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:46.505684 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqq6d" event={"ID":"595ebd05-e643-4645-9025-b3bddb2e02db","Type":"ContainerStarted","Data":"d666d5a845a95b83f2bd07e40a7a66d4a9f9b43aa4d0962d6816345deb269a29"} Jul 14 19:47:46.506217 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:46.505737 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqq6d" event={"ID":"595ebd05-e643-4645-9025-b3bddb2e02db","Type":"ContainerStarted","Data":"90222c4442d253018c0040e60d156067b5ab84da27bc3726b8c6f95dd32b9087"} Jul 14 19:47:46.527765 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:46.527706 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tqq6d" podStartSLOduration=2.694829545 podStartE2EDuration="4.527687387s" totalImagesPullingTime="1.722181293s" totalInitContainerRuntime="110.676549ms" isStatefulPod=true podCreationTimestamp="2026-07-14 19:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:47:46.525808858 +0000 UTC m=+108.173372425" watchObservedRunningTime="2026-07-14 19:47:46.527687387 +0000 UTC m=+108.175250952" Jul 14 19:47:59.562138 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:59.562101 2578 generic.go:356] "Generic (PLEG): container finished" podID="4c335c34-95b3-4b24-8b1c-0e503876aeb9" containerID="43b3931d9945ab0cf312aa15660dfccc8456efa5d351b7ca1a995d0b8a504be4" exitCode=0 Jul 14 19:47:59.562622 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:59.562189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" event={"ID":"4c335c34-95b3-4b24-8b1c-0e503876aeb9","Type":"ContainerDied","Data":"43b3931d9945ab0cf312aa15660dfccc8456efa5d351b7ca1a995d0b8a504be4"} Jul 14 19:47:59.562622 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:47:59.562534 2578 scope.go:117] "RemoveContainer" containerID="43b3931d9945ab0cf312aa15660dfccc8456efa5d351b7ca1a995d0b8a504be4" Jul 14 19:48:00.328143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.328098 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" podUID="c77be464-74ee-490f-aed1-99bed17482ef" containerName="registry" containerID="cri-o://93d1196253104bc645a59d55a085637aad4bc15c82a59a2e4444cde85b5f695e" gracePeriod=30 Jul 14 19:48:00.569338 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.569303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-795zv" event={"ID":"4c335c34-95b3-4b24-8b1c-0e503876aeb9","Type":"ContainerStarted","Data":"74e71a000b3934ae4cd5c4cdad23a8a2bf57e4b03bc7f588545bd4419dda35d7"} Jul 14 19:48:00.570686 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.570654 2578 generic.go:356] "Generic (PLEG): container finished" podID="c77be464-74ee-490f-aed1-99bed17482ef" containerID="93d1196253104bc645a59d55a085637aad4bc15c82a59a2e4444cde85b5f695e" exitCode=0 Jul 14 19:48:00.570788 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.570721 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" event={"ID":"c77be464-74ee-490f-aed1-99bed17482ef","Type":"ContainerDied","Data":"93d1196253104bc645a59d55a085637aad4bc15c82a59a2e4444cde85b5f695e"} Jul 14 19:48:00.570788 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.570766 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" event={"ID":"c77be464-74ee-490f-aed1-99bed17482ef","Type":"ContainerDied","Data":"3b1250755a742f42674068c13fc3a1ff60c4840892dfe29e06d200ecfa805c67"} Jul 14 19:48:00.570788 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.570780 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b1250755a742f42674068c13fc3a1ff60c4840892dfe29e06d200ecfa805c67" Jul 14 19:48:00.577348 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.577329 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:48:00.656130 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.656042 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77be464-74ee-490f-aed1-99bed17482ef-ca-trust-extracted\") pod \"c77be464-74ee-490f-aed1-99bed17482ef\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " Jul 14 19:48:00.656130 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.656082 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") pod \"c77be464-74ee-490f-aed1-99bed17482ef\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " Jul 14 19:48:00.658360 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.658327 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c77be464-74ee-490f-aed1-99bed17482ef" (UID: "c77be464-74ee-490f-aed1-99bed17482ef"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:48:00.663578 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.663553 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77be464-74ee-490f-aed1-99bed17482ef-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c77be464-74ee-490f-aed1-99bed17482ef" (UID: "c77be464-74ee-490f-aed1-99bed17482ef"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:48:00.757226 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757191 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-image-registry-private-configuration\") pod \"c77be464-74ee-490f-aed1-99bed17482ef\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " Jul 14 19:48:00.757406 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757232 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-registry-certificates\") pod \"c77be464-74ee-490f-aed1-99bed17482ef\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " Jul 14 19:48:00.757406 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757257 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-installation-pull-secrets\") pod \"c77be464-74ee-490f-aed1-99bed17482ef\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " Jul 14 19:48:00.757406 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757318 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-kube-api-access-cpq4q\") pod \"c77be464-74ee-490f-aed1-99bed17482ef\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " Jul 14 19:48:00.757406 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757340 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-bound-sa-token\") pod \"c77be464-74ee-490f-aed1-99bed17482ef\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " Jul 14 19:48:00.757406 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757395 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-trusted-ca\") pod \"c77be464-74ee-490f-aed1-99bed17482ef\" (UID: \"c77be464-74ee-490f-aed1-99bed17482ef\") " Jul 14 19:48:00.757647 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757628 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-registry-tls\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:48:00.757698 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757645 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77be464-74ee-490f-aed1-99bed17482ef-ca-trust-extracted\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:48:00.757799 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757761 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c77be464-74ee-490f-aed1-99bed17482ef" (UID: "c77be464-74ee-490f-aed1-99bed17482ef"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:00.757951 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.757875 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c77be464-74ee-490f-aed1-99bed17482ef" (UID: "c77be464-74ee-490f-aed1-99bed17482ef"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:48:00.759570 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.759543 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c77be464-74ee-490f-aed1-99bed17482ef" (UID: "c77be464-74ee-490f-aed1-99bed17482ef"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:48:00.759667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.759551 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c77be464-74ee-490f-aed1-99bed17482ef" (UID: "c77be464-74ee-490f-aed1-99bed17482ef"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:00.759667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.759649 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-kube-api-access-cpq4q" (OuterVolumeSpecName: "kube-api-access-cpq4q") pod "c77be464-74ee-490f-aed1-99bed17482ef" (UID: "c77be464-74ee-490f-aed1-99bed17482ef"). InnerVolumeSpecName "kube-api-access-cpq4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:48:00.759740 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.759656 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c77be464-74ee-490f-aed1-99bed17482ef" (UID: "c77be464-74ee-490f-aed1-99bed17482ef"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:48:00.858636 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.858600 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-trusted-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:48:00.858636 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.858632 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-image-registry-private-configuration\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:48:00.858636 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.858644 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77be464-74ee-490f-aed1-99bed17482ef-registry-certificates\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:48:00.858916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.858654 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77be464-74ee-490f-aed1-99bed17482ef-installation-pull-secrets\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:48:00.858916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.858663 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpq4q\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-kube-api-access-cpq4q\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:48:00.858916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:00.858673 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77be464-74ee-490f-aed1-99bed17482ef-bound-sa-token\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:48:01.574914 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:01.574882 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64cdb65b65-wgdv9" Jul 14 19:48:01.593006 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:01.592974 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64cdb65b65-wgdv9"] Jul 14 19:48:01.599113 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:01.599084 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64cdb65b65-wgdv9"] Jul 14 19:48:02.882410 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:02.882370 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c77be464-74ee-490f-aed1-99bed17482ef" path="/var/lib/kubelet/pods/c77be464-74ee-490f-aed1-99bed17482ef/volumes" Jul 14 19:48:08.604445 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:08.604414 2578 generic.go:356] "Generic (PLEG): container finished" podID="5d04d06d-c61d-4982-895f-50add148475a" containerID="a6587e146d1d09869922223b06dae1d2b7d9e774520a42a2804310b50a93f072" exitCode=0 Jul 14 19:48:08.604810 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:08.604483 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" event={"ID":"5d04d06d-c61d-4982-895f-50add148475a","Type":"ContainerDied","Data":"a6587e146d1d09869922223b06dae1d2b7d9e774520a42a2804310b50a93f072"} Jul 14 19:48:08.604810 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:08.604802 2578 scope.go:117] "RemoveContainer" containerID="a6587e146d1d09869922223b06dae1d2b7d9e774520a42a2804310b50a93f072" Jul 14 19:48:09.609719 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:09.609680 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-pzbmv" event={"ID":"5d04d06d-c61d-4982-895f-50add148475a","Type":"ContainerStarted","Data":"a302f3eb89f0b00f20844f2cfc70b6e9143f42851a25f7ccc23bf15187e88032"} Jul 14 19:48:14.632407 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:14.632370 2578 generic.go:356] "Generic (PLEG): container finished" podID="85b2851b-b372-4935-9b8b-379379b4ae92" containerID="7e3c875ec0e17eee1e93e78184ce008f399c84bb454a355e9c29ee2a7581bd05" exitCode=0 Jul 14 19:48:14.632797 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:14.632451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" event={"ID":"85b2851b-b372-4935-9b8b-379379b4ae92","Type":"ContainerDied","Data":"7e3c875ec0e17eee1e93e78184ce008f399c84bb454a355e9c29ee2a7581bd05"} Jul 14 19:48:14.632887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:14.632873 2578 scope.go:117] "RemoveContainer" containerID="7e3c875ec0e17eee1e93e78184ce008f399c84bb454a355e9c29ee2a7581bd05" Jul 14 19:48:15.638536 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:48:15.638495 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-6wwg6" event={"ID":"85b2851b-b372-4935-9b8b-379379b4ae92","Type":"ContainerStarted","Data":"e35b4951a97b950a0028c51044d5a4770c79061360466383e9aa5af122a37c8a"} Jul 14 19:50:58.775988 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:50:58.775954 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 19:50:58.778663 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:50:58.778641 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 19:50:58.783326 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:50:58.783302 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:50:58.786178 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:50:58.786152 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:50:58.787210 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:50:58.787189 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 19:51:26.403814 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.403782 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87"] Jul 14 19:51:26.405086 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.404106 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77be464-74ee-490f-aed1-99bed17482ef" containerName="registry" Jul 14 19:51:26.405086 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.404118 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77be464-74ee-490f-aed1-99bed17482ef" containerName="registry" Jul 14 19:51:26.405086 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.404173 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77be464-74ee-490f-aed1-99bed17482ef" containerName="registry" Jul 14 19:51:26.406052 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.406037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.409799 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.409771 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 19:51:26.409970 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.409861 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 19:51:26.409970 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.409879 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5l9cj\"" Jul 14 19:51:26.410081 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.410071 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 19:51:26.410126 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.410071 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 19:51:26.422008 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.421982 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87"] Jul 14 19:51:26.583231 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.583194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd42ad44-f326-448d-8795-71faf345277d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.583444 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.583246 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd42ad44-f326-448d-8795-71faf345277d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.583444 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.583318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd42ad44-f326-448d-8795-71faf345277d-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.583444 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.583398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd42ad44-f326-448d-8795-71faf345277d-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.583444 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.583414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhjx\" (UniqueName: \"kubernetes.io/projected/fd42ad44-f326-448d-8795-71faf345277d-kube-api-access-ghhjx\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.684754 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.684671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd42ad44-f326-448d-8795-71faf345277d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.684754 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.684720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd42ad44-f326-448d-8795-71faf345277d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.684974 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.684864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd42ad44-f326-448d-8795-71faf345277d-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.685036 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.684972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd42ad44-f326-448d-8795-71faf345277d-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.685036 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.685005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhjx\" (UniqueName: \"kubernetes.io/projected/fd42ad44-f326-448d-8795-71faf345277d-kube-api-access-ghhjx\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.685249 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.685219 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd42ad44-f326-448d-8795-71faf345277d-manifests\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.687130 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.687096 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd42ad44-f326-448d-8795-71faf345277d-tmp\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.687228 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.687182 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd42ad44-f326-448d-8795-71faf345277d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.687228 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.687191 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd42ad44-f326-448d-8795-71faf345277d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.695438 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.695415 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhjx\" (UniqueName: \"kubernetes.io/projected/fd42ad44-f326-448d-8795-71faf345277d-kube-api-access-ghhjx\") pod \"opendatahub-operator-controller-manager-7f59db9768-mkd87\" (UID: \"fd42ad44-f326-448d-8795-71faf345277d\") " pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.716486 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.716467 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:26.855539 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.855505 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87"] Jul 14 19:51:26.859122 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:51:26.859097 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd42ad44_f326_448d_8795_71faf345277d.slice/crio-c4653b9faa6ebe6d16d670ac701130b8bf061489024dddcc28d5c46ee471cb71 WatchSource:0}: Error finding container c4653b9faa6ebe6d16d670ac701130b8bf061489024dddcc28d5c46ee471cb71: Status 404 returned error can't find the container with id c4653b9faa6ebe6d16d670ac701130b8bf061489024dddcc28d5c46ee471cb71 Jul 14 19:51:26.860856 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:26.860817 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:51:27.379305 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:27.379269 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" event={"ID":"fd42ad44-f326-448d-8795-71faf345277d","Type":"ContainerStarted","Data":"c4653b9faa6ebe6d16d670ac701130b8bf061489024dddcc28d5c46ee471cb71"} Jul 14 19:51:30.395214 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:30.395171 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" event={"ID":"fd42ad44-f326-448d-8795-71faf345277d","Type":"ContainerStarted","Data":"f4b1ae2497048b850946ddfee6bbffdcc6815e1010b7a18bdb0bf36e147fb1cd"} Jul 14 19:51:31.401588 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:31.401488 2578 generic.go:356] "Generic (PLEG): container finished" podID="fd42ad44-f326-448d-8795-71faf345277d" containerID="f4b1ae2497048b850946ddfee6bbffdcc6815e1010b7a18bdb0bf36e147fb1cd" exitCode=0 Jul 14 19:51:31.401588 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:31.401542 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" event={"ID":"fd42ad44-f326-448d-8795-71faf345277d","Type":"ContainerDied","Data":"f4b1ae2497048b850946ddfee6bbffdcc6815e1010b7a18bdb0bf36e147fb1cd"} Jul 14 19:51:33.411495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:33.411450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" event={"ID":"fd42ad44-f326-448d-8795-71faf345277d","Type":"ContainerStarted","Data":"478eb5b66ef312e221dfe879e300089f3ac52f87d8ef23f146dc455264c9c458"} Jul 14 19:51:33.412112 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:33.412090 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:33.436909 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:33.436858 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" podStartSLOduration=2.108641704 podStartE2EDuration="7.436843591s" totalImagesPullingTime="4.251521582s" totalInitContainerRuntime="1.076680305s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:33.435622727 +0000 UTC m=+335.083186318" watchObservedRunningTime="2026-07-14 19:51:33.436843591 +0000 UTC m=+335.084407156" Jul 14 19:51:44.417757 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:44.417721 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f59db9768-mkd87" Jul 14 19:51:48.016984 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.016945 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rgc5q"] Jul 14 19:51:48.025838 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.025801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:48.028528 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.028508 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 14 19:51:48.028635 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.028579 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-76dvt\"" Jul 14 19:51:48.034311 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.034287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rgc5q"] Jul 14 19:51:48.046406 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.046377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-cert\") pod \"odh-model-controller-7998c67dc4-rgc5q\" (UID: \"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4\") " pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:48.046546 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.046453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvf86\" (UniqueName: \"kubernetes.io/projected/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-kube-api-access-gvf86\") pod \"odh-model-controller-7998c67dc4-rgc5q\" (UID: \"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4\") " pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:48.147756 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.147724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-cert\") pod \"odh-model-controller-7998c67dc4-rgc5q\" (UID: \"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4\") " pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:48.147756 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.147761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvf86\" (UniqueName: \"kubernetes.io/projected/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-kube-api-access-gvf86\") pod \"odh-model-controller-7998c67dc4-rgc5q\" (UID: \"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4\") " pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:48.147993 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:48.147905 2578 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 14 19:51:48.147993 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:48.147984 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-cert podName:0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4 nodeName:}" failed. No retries permitted until 2026-07-14 19:51:48.647966102 +0000 UTC m=+350.295529649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-cert") pod "odh-model-controller-7998c67dc4-rgc5q" (UID: "0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4") : secret "odh-model-controller-webhook-cert" not found Jul 14 19:51:48.157449 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.157424 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvf86\" (UniqueName: \"kubernetes.io/projected/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-kube-api-access-gvf86\") pod \"odh-model-controller-7998c67dc4-rgc5q\" (UID: \"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4\") " pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:48.651694 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.651654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-cert\") pod \"odh-model-controller-7998c67dc4-rgc5q\" (UID: \"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4\") " pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:48.654115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.654091 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4-cert\") pod \"odh-model-controller-7998c67dc4-rgc5q\" (UID: \"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4\") " pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:48.699905 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.699872 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bcd84"] Jul 14 19:51:48.702533 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.702515 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:48.704862 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.704823 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 14 19:51:48.714391 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.714364 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bcd84"] Jul 14 19:51:48.752887 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.752822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a97c62-1a40-4ab4-b6be-25580d851593-tls-certs\") pod \"model-serving-api-54d89bd5fc-bcd84\" (UID: \"d8a97c62-1a40-4ab4-b6be-25580d851593\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:48.753069 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.752928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzpk\" (UniqueName: \"kubernetes.io/projected/d8a97c62-1a40-4ab4-b6be-25580d851593-kube-api-access-rbzpk\") pod \"model-serving-api-54d89bd5fc-bcd84\" (UID: \"d8a97c62-1a40-4ab4-b6be-25580d851593\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:48.853695 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.853640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzpk\" (UniqueName: \"kubernetes.io/projected/d8a97c62-1a40-4ab4-b6be-25580d851593-kube-api-access-rbzpk\") pod \"model-serving-api-54d89bd5fc-bcd84\" (UID: \"d8a97c62-1a40-4ab4-b6be-25580d851593\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:48.853910 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.853750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a97c62-1a40-4ab4-b6be-25580d851593-tls-certs\") pod \"model-serving-api-54d89bd5fc-bcd84\" (UID: \"d8a97c62-1a40-4ab4-b6be-25580d851593\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:48.853910 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:48.853887 2578 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 14 19:51:48.854043 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:48.853953 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a97c62-1a40-4ab4-b6be-25580d851593-tls-certs podName:d8a97c62-1a40-4ab4-b6be-25580d851593 nodeName:}" failed. No retries permitted until 2026-07-14 19:51:49.353933197 +0000 UTC m=+351.001496741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d8a97c62-1a40-4ab4-b6be-25580d851593-tls-certs") pod "model-serving-api-54d89bd5fc-bcd84" (UID: "d8a97c62-1a40-4ab4-b6be-25580d851593") : secret "model-serving-api-tls" not found Jul 14 19:51:48.862903 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.862874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzpk\" (UniqueName: \"kubernetes.io/projected/d8a97c62-1a40-4ab4-b6be-25580d851593-kube-api-access-rbzpk\") pod \"model-serving-api-54d89bd5fc-bcd84\" (UID: \"d8a97c62-1a40-4ab4-b6be-25580d851593\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:48.936535 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:48.936434 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:49.081841 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:49.081803 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rgc5q"] Jul 14 19:51:49.083701 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:51:49.083668 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fcb2f4e_84a6_4aa3_a585_0c3318a97bf4.slice/crio-c82a988ff1cc144bdcd5a691036aea34d4737ab6c7d3a07f5502b8f73cece42e WatchSource:0}: Error finding container c82a988ff1cc144bdcd5a691036aea34d4737ab6c7d3a07f5502b8f73cece42e: Status 404 returned error can't find the container with id c82a988ff1cc144bdcd5a691036aea34d4737ab6c7d3a07f5502b8f73cece42e Jul 14 19:51:49.357318 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:49.357273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a97c62-1a40-4ab4-b6be-25580d851593-tls-certs\") pod \"model-serving-api-54d89bd5fc-bcd84\" (UID: \"d8a97c62-1a40-4ab4-b6be-25580d851593\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:49.359709 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:49.359677 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8a97c62-1a40-4ab4-b6be-25580d851593-tls-certs\") pod \"model-serving-api-54d89bd5fc-bcd84\" (UID: \"d8a97c62-1a40-4ab4-b6be-25580d851593\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:49.478631 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:49.478596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" event={"ID":"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4","Type":"ContainerStarted","Data":"c82a988ff1cc144bdcd5a691036aea34d4737ab6c7d3a07f5502b8f73cece42e"} Jul 14 19:51:49.614340 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:49.614247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:49.742601 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:49.742555 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bcd84"] Jul 14 19:51:49.744665 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:51:49.744636 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a97c62_1a40_4ab4_b6be_25580d851593.slice/crio-80b0deb18faf019b574637ff4474ae65b5c65ebe3167c247f0610f963e8ca1e8 WatchSource:0}: Error finding container 80b0deb18faf019b574637ff4474ae65b5c65ebe3167c247f0610f963e8ca1e8: Status 404 returned error can't find the container with id 80b0deb18faf019b574637ff4474ae65b5c65ebe3167c247f0610f963e8ca1e8 Jul 14 19:51:50.484619 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:50.484580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" event={"ID":"d8a97c62-1a40-4ab4-b6be-25580d851593","Type":"ContainerStarted","Data":"80b0deb18faf019b574637ff4474ae65b5c65ebe3167c247f0610f963e8ca1e8"} Jul 14 19:51:52.495744 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:52.495665 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" event={"ID":"d8a97c62-1a40-4ab4-b6be-25580d851593","Type":"ContainerStarted","Data":"dfd3ea423a595d712c03c73c4b085c7a6c3a4d16cdc781e3b1e979e46ed047ca"} Jul 14 19:51:52.496185 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:52.495779 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:51:52.513308 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:52.513251 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" podStartSLOduration=2.089170111 podStartE2EDuration="4.513233212s" totalImagesPullingTime="2.424063101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:52.512609529 +0000 UTC m=+354.160173099" watchObservedRunningTime="2026-07-14 19:51:52.513233212 +0000 UTC m=+354.160796780" Jul 14 19:51:53.501704 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:53.501665 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" event={"ID":"0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4","Type":"ContainerStarted","Data":"caac55edbda7a6cca875986b8b606386125fcca5fa6e807397b5061c8309d6d2"} Jul 14 19:51:53.502214 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:53.501966 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:51:53.520915 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:53.520858 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" podStartSLOduration=2.588739581 podStartE2EDuration="6.520820176s" totalImagesPullingTime="3.932080595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:51:53.520587705 +0000 UTC m=+355.168151272" watchObservedRunningTime="2026-07-14 19:51:53.520820176 +0000 UTC m=+355.168383743" Jul 14 19:51:57.401261 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.401223 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm"] Jul 14 19:51:57.409679 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.409649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:57.411775 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.411749 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg"] Jul 14 19:51:57.413607 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.413588 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 14 19:51:57.413959 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.413942 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qzpmw\"" Jul 14 19:51:57.415295 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.415278 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:57.417808 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.417783 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 14 19:51:57.417808 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.417799 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7ldkx\"" Jul 14 19:51:57.420964 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.420938 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm"] Jul 14 19:51:57.425889 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.425866 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg"] Jul 14 19:51:57.524037 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.524000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847tt\" (UniqueName: \"kubernetes.io/projected/99de2193-68ea-4c3f-bf6a-efa88689502a-kube-api-access-847tt\") pod \"kserve-controller-manager-d6f4b65b8-lc6qm\" (UID: \"99de2193-68ea-4c3f-bf6a-efa88689502a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:57.524214 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.524080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1674e59-0ac4-4f18-8047-4de1c2793908-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fh5pg\" (UID: \"f1674e59-0ac4-4f18-8047-4de1c2793908\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:57.524214 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.524144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vtk\" (UniqueName: \"kubernetes.io/projected/f1674e59-0ac4-4f18-8047-4de1c2793908-kube-api-access-t7vtk\") pod \"llmisvc-controller-manager-777c6bd7d5-fh5pg\" (UID: \"f1674e59-0ac4-4f18-8047-4de1c2793908\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:57.524214 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.524182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99de2193-68ea-4c3f-bf6a-efa88689502a-cert\") pod \"kserve-controller-manager-d6f4b65b8-lc6qm\" (UID: \"99de2193-68ea-4c3f-bf6a-efa88689502a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:57.625303 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.625262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847tt\" (UniqueName: \"kubernetes.io/projected/99de2193-68ea-4c3f-bf6a-efa88689502a-kube-api-access-847tt\") pod \"kserve-controller-manager-d6f4b65b8-lc6qm\" (UID: \"99de2193-68ea-4c3f-bf6a-efa88689502a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:57.625516 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.625330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1674e59-0ac4-4f18-8047-4de1c2793908-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fh5pg\" (UID: \"f1674e59-0ac4-4f18-8047-4de1c2793908\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:57.625516 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.625362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vtk\" (UniqueName: \"kubernetes.io/projected/f1674e59-0ac4-4f18-8047-4de1c2793908-kube-api-access-t7vtk\") pod \"llmisvc-controller-manager-777c6bd7d5-fh5pg\" (UID: \"f1674e59-0ac4-4f18-8047-4de1c2793908\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:57.625516 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.625376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99de2193-68ea-4c3f-bf6a-efa88689502a-cert\") pod \"kserve-controller-manager-d6f4b65b8-lc6qm\" (UID: \"99de2193-68ea-4c3f-bf6a-efa88689502a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:57.625697 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:57.625526 2578 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 14 19:51:57.625697 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:57.625526 2578 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 14 19:51:57.625697 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:57.625619 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1674e59-0ac4-4f18-8047-4de1c2793908-cert podName:f1674e59-0ac4-4f18-8047-4de1c2793908 nodeName:}" failed. No retries permitted until 2026-07-14 19:51:58.125600118 +0000 UTC m=+359.773163665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f1674e59-0ac4-4f18-8047-4de1c2793908-cert") pod "llmisvc-controller-manager-777c6bd7d5-fh5pg" (UID: "f1674e59-0ac4-4f18-8047-4de1c2793908") : secret "llmisvc-webhook-server-cert" not found Jul 14 19:51:57.625697 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:57.625675 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99de2193-68ea-4c3f-bf6a-efa88689502a-cert podName:99de2193-68ea-4c3f-bf6a-efa88689502a nodeName:}" failed. No retries permitted until 2026-07-14 19:51:58.125655165 +0000 UTC m=+359.773218723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/99de2193-68ea-4c3f-bf6a-efa88689502a-cert") pod "kserve-controller-manager-d6f4b65b8-lc6qm" (UID: "99de2193-68ea-4c3f-bf6a-efa88689502a") : secret "kserve-webhook-server-cert" not found Jul 14 19:51:57.635509 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.635482 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-847tt\" (UniqueName: \"kubernetes.io/projected/99de2193-68ea-4c3f-bf6a-efa88689502a-kube-api-access-847tt\") pod \"kserve-controller-manager-d6f4b65b8-lc6qm\" (UID: \"99de2193-68ea-4c3f-bf6a-efa88689502a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:57.635632 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:57.635542 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vtk\" (UniqueName: \"kubernetes.io/projected/f1674e59-0ac4-4f18-8047-4de1c2793908-kube-api-access-t7vtk\") pod \"llmisvc-controller-manager-777c6bd7d5-fh5pg\" (UID: \"f1674e59-0ac4-4f18-8047-4de1c2793908\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:58.130486 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.130450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1674e59-0ac4-4f18-8047-4de1c2793908-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fh5pg\" (UID: \"f1674e59-0ac4-4f18-8047-4de1c2793908\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:58.130486 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.130492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99de2193-68ea-4c3f-bf6a-efa88689502a-cert\") pod \"kserve-controller-manager-d6f4b65b8-lc6qm\" (UID: \"99de2193-68ea-4c3f-bf6a-efa88689502a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:58.132942 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.132918 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1674e59-0ac4-4f18-8047-4de1c2793908-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-fh5pg\" (UID: \"f1674e59-0ac4-4f18-8047-4de1c2793908\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:58.133076 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.132967 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99de2193-68ea-4c3f-bf6a-efa88689502a-cert\") pod \"kserve-controller-manager-d6f4b65b8-lc6qm\" (UID: \"99de2193-68ea-4c3f-bf6a-efa88689502a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:58.212270 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.212230 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7"] Jul 14 19:51:58.216072 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.216056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.221980 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.221955 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 19:51:58.221980 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.221971 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 19:51:58.222171 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.221979 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hl25t\"" Jul 14 19:51:58.229566 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.229539 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7"] Jul 14 19:51:58.321396 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.321359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:51:58.332165 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.332136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e58882e-0431-4140-9998-3303528d1457-tmp\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.332298 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.332258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slf4t\" (UniqueName: \"kubernetes.io/projected/2e58882e-0431-4140-9998-3303528d1457-kube-api-access-slf4t\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.332355 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.332328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e58882e-0431-4140-9998-3303528d1457-tls-certs\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.335012 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.334984 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:51:58.434051 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.434022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf4t\" (UniqueName: \"kubernetes.io/projected/2e58882e-0431-4140-9998-3303528d1457-kube-api-access-slf4t\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.434445 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.434081 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e58882e-0431-4140-9998-3303528d1457-tls-certs\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.434445 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.434112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e58882e-0431-4140-9998-3303528d1457-tmp\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.434445 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:58.434247 2578 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 14 19:51:58.434445 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:51:58.434307 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e58882e-0431-4140-9998-3303528d1457-tls-certs podName:2e58882e-0431-4140-9998-3303528d1457 nodeName:}" failed. No retries permitted until 2026-07-14 19:51:58.934286894 +0000 UTC m=+360.581850437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2e58882e-0431-4140-9998-3303528d1457-tls-certs") pod "kube-auth-proxy-5df66fb8c-jh5t7" (UID: "2e58882e-0431-4140-9998-3303528d1457") : secret "kube-auth-proxy-tls" not found Jul 14 19:51:58.437033 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.437009 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e58882e-0431-4140-9998-3303528d1457-tmp\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.462191 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.462157 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slf4t\" (UniqueName: \"kubernetes.io/projected/2e58882e-0431-4140-9998-3303528d1457-kube-api-access-slf4t\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.482078 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.482040 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm"] Jul 14 19:51:58.485449 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:51:58.485425 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99de2193_68ea_4c3f_bf6a_efa88689502a.slice/crio-eba0bb7edd36f99dde48d7753001f9b144b0402b3d16de64b78b178d5ccf4094 WatchSource:0}: Error finding container eba0bb7edd36f99dde48d7753001f9b144b0402b3d16de64b78b178d5ccf4094: Status 404 returned error can't find the container with id eba0bb7edd36f99dde48d7753001f9b144b0402b3d16de64b78b178d5ccf4094 Jul 14 19:51:58.496395 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:51:58.496373 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1674e59_0ac4_4f18_8047_4de1c2793908.slice/crio-c669bae20ad3f54c332ed775b0d90b7aab776f082dab49a870a9888c952b27e7 WatchSource:0}: Error finding container c669bae20ad3f54c332ed775b0d90b7aab776f082dab49a870a9888c952b27e7: Status 404 returned error can't find the container with id c669bae20ad3f54c332ed775b0d90b7aab776f082dab49a870a9888c952b27e7 Jul 14 19:51:58.500047 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.500027 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg"] Jul 14 19:51:58.526853 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.526800 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" event={"ID":"f1674e59-0ac4-4f18-8047-4de1c2793908","Type":"ContainerStarted","Data":"c669bae20ad3f54c332ed775b0d90b7aab776f082dab49a870a9888c952b27e7"} Jul 14 19:51:58.527886 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.527865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" event={"ID":"99de2193-68ea-4c3f-bf6a-efa88689502a","Type":"ContainerStarted","Data":"eba0bb7edd36f99dde48d7753001f9b144b0402b3d16de64b78b178d5ccf4094"} Jul 14 19:51:58.937782 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.937748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e58882e-0431-4140-9998-3303528d1457-tls-certs\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:58.940325 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:58.940296 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e58882e-0431-4140-9998-3303528d1457-tls-certs\") pod \"kube-auth-proxy-5df66fb8c-jh5t7\" (UID: \"2e58882e-0431-4140-9998-3303528d1457\") " pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:59.129040 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:59.129009 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hl25t\"" Jul 14 19:51:59.137256 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:59.137231 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" Jul 14 19:51:59.344070 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:59.344036 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7"] Jul 14 19:51:59.411138 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:51:59.411107 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e58882e_0431_4140_9998_3303528d1457.slice/crio-02cbff04b6952bc71011348f725dc18f7df31a17ad25909453f8e752d0e55d0d WatchSource:0}: Error finding container 02cbff04b6952bc71011348f725dc18f7df31a17ad25909453f8e752d0e55d0d: Status 404 returned error can't find the container with id 02cbff04b6952bc71011348f725dc18f7df31a17ad25909453f8e752d0e55d0d Jul 14 19:51:59.535344 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:51:59.535260 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" event={"ID":"2e58882e-0431-4140-9998-3303528d1457","Type":"ContainerStarted","Data":"02cbff04b6952bc71011348f725dc18f7df31a17ad25909453f8e752d0e55d0d"} Jul 14 19:52:01.547124 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:01.547055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" event={"ID":"99de2193-68ea-4c3f-bf6a-efa88689502a","Type":"ContainerStarted","Data":"0dcb3f486c3b2ddf91fb8f44caa9a12b0c35ffd63fc6b2d563f134c5e2858086"} Jul 14 19:52:01.578308 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:01.578242 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" podStartSLOduration=1.676970157 podStartE2EDuration="4.578218994s" totalImagesPullingTime="2.901248837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:01.575601305 +0000 UTC m=+363.223164871" watchObservedRunningTime="2026-07-14 19:52:01.578218994 +0000 UTC m=+363.225782559" Jul 14 19:52:02.554578 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:02.554457 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" event={"ID":"f1674e59-0ac4-4f18-8047-4de1c2793908","Type":"ContainerStarted","Data":"df3b63c81735e2839210c7e8962d0257967869be914c22efb0f74520779705b0"} Jul 14 19:52:02.555009 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:02.554583 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:52:02.573726 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:02.573663 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" podStartSLOduration=1.964651221 podStartE2EDuration="5.573647987s" totalImagesPullingTime="3.608996766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:02.571535267 +0000 UTC m=+364.219098833" watchObservedRunningTime="2026-07-14 19:52:02.573647987 +0000 UTC m=+364.221211552" Jul 14 19:52:03.507007 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:03.506978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bcd84" Jul 14 19:52:03.560483 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:03.560435 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:52:04.508382 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.508344 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rgc5q" Jul 14 19:52:04.518289 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.518264 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 19:52:04.730792 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.730758 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n"] Jul 14 19:52:04.734769 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.734738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.738499 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.737961 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xlcvq\"" Jul 14 19:52:04.738499 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.738224 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 19:52:04.751178 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.751143 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n"] Jul 14 19:52:04.791678 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ndwz\" (UniqueName: \"kubernetes.io/projected/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-kube-api-access-9ndwz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.791678 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.791921 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.791921 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.791921 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.791921 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.791921 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.792075 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.792075 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.791958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893160 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ndwz\" (UniqueName: \"kubernetes.io/projected/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-kube-api-access-9ndwz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893348 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893348 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893348 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893348 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893348 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893348 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893630 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893630 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893818 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893739 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.893818 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.893773 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.894102 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.894072 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.894274 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.894098 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.894378 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.894354 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.896109 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.896084 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.896109 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.896103 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.903933 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.903906 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:04.904122 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:04.904101 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ndwz\" (UniqueName: \"kubernetes.io/projected/9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c-kube-api-access-9ndwz\") pod \"data-science-gateway-data-science-gateway-class-6576859cf494s6n\" (UID: \"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:05.048753 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:05.048664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:05.200368 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:05.200340 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n"] Jul 14 19:52:05.201709 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:52:05.201667 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef901b7_0fdb_4cc5_9ec8_4908f555cd2c.slice/crio-1c8f687646183d83e498c6babadc8611972f066d5c8ee43fe67e447a946f51b7 WatchSource:0}: Error finding container 1c8f687646183d83e498c6babadc8611972f066d5c8ee43fe67e447a946f51b7: Status 404 returned error can't find the container with id 1c8f687646183d83e498c6babadc8611972f066d5c8ee43fe67e447a946f51b7 Jul 14 19:52:05.574005 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:05.573966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" event={"ID":"2e58882e-0431-4140-9998-3303528d1457","Type":"ContainerStarted","Data":"233e858491f1f30d6f36b104cf233e16be23aadd594ef6a0feaad4d71f4cfeca"} Jul 14 19:52:05.575143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:05.575112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" event={"ID":"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c","Type":"ContainerStarted","Data":"1c8f687646183d83e498c6babadc8611972f066d5c8ee43fe67e447a946f51b7"} Jul 14 19:52:05.595736 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:05.595688 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5df66fb8c-jh5t7" podStartSLOduration=2.492931954 podStartE2EDuration="7.595675131s" totalImagesPullingTime="5.102743177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:05.594547458 +0000 UTC m=+367.242111023" watchObservedRunningTime="2026-07-14 19:52:05.595675131 +0000 UTC m=+367.243238696" Jul 14 19:52:08.090586 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:08.090546 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:08.090845 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:08.090625 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:08.090845 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:08.090654 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:52:08.592067 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:08.592033 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" event={"ID":"9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c","Type":"ContainerStarted","Data":"8dffcde48eb4056af3038d2126a05ed4e76d561c89d00ae999bfc332b77aa39c"} Jul 14 19:52:08.616149 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:08.616099 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" podStartSLOduration=1.729426615 podStartE2EDuration="4.616083889s" totalImagesPullingTime="2.886657274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:08.615102177 +0000 UTC m=+370.262665790" watchObservedRunningTime="2026-07-14 19:52:08.616083889 +0000 UTC m=+370.263647454" Jul 14 19:52:09.049564 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:09.049475 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:09.142127 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:09.142092 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:09.596903 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:09.596866 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:09.601856 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:09.598408 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf494s6n" Jul 14 19:52:33.566047 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:33.566016 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lc6qm" Jul 14 19:52:34.568897 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:34.568864 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-fh5pg" Jul 14 19:52:48.459429 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.459394 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kxrdf"] Jul 14 19:52:48.470182 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.470153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" Jul 14 19:52:48.474265 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.474060 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-gtx8f\"" Jul 14 19:52:48.474265 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.474080 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 19:52:48.474265 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.474187 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 19:52:48.475310 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.475289 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kxrdf"] Jul 14 19:52:48.569228 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.569193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgwb\" (UniqueName: \"kubernetes.io/projected/f76c70df-48c4-403f-9d58-0ba6fab78c6b-kube-api-access-wpgwb\") pod \"kuadrant-operator-catalog-kxrdf\" (UID: \"f76c70df-48c4-403f-9d58-0ba6fab78c6b\") " pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" Jul 14 19:52:48.670394 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.670357 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgwb\" (UniqueName: \"kubernetes.io/projected/f76c70df-48c4-403f-9d58-0ba6fab78c6b-kube-api-access-wpgwb\") pod \"kuadrant-operator-catalog-kxrdf\" (UID: \"f76c70df-48c4-403f-9d58-0ba6fab78c6b\") " pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" Jul 14 19:52:48.679818 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.679798 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgwb\" (UniqueName: \"kubernetes.io/projected/f76c70df-48c4-403f-9d58-0ba6fab78c6b-kube-api-access-wpgwb\") pod \"kuadrant-operator-catalog-kxrdf\" (UID: \"f76c70df-48c4-403f-9d58-0ba6fab78c6b\") " pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" Jul 14 19:52:48.782879 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.782779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" Jul 14 19:52:48.826549 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.826515 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kxrdf"] Jul 14 19:52:48.931713 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:48.931685 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kxrdf"] Jul 14 19:52:48.935134 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:52:48.935108 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76c70df_48c4_403f_9d58_0ba6fab78c6b.slice/crio-c11045d773f3b1502938257f602f3d3ac1243553c7fee909541ccae18996ef13 WatchSource:0}: Error finding container c11045d773f3b1502938257f602f3d3ac1243553c7fee909541ccae18996ef13: Status 404 returned error can't find the container with id c11045d773f3b1502938257f602f3d3ac1243553c7fee909541ccae18996ef13 Jul 14 19:52:49.036156 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.036075 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnc6t"] Jul 14 19:52:49.040477 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.040441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:52:49.046342 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.046314 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnc6t"] Jul 14 19:52:49.076432 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.076396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhnhq\" (UniqueName: \"kubernetes.io/projected/29843439-dd60-4557-a0b9-6db34a36a6a9-kube-api-access-rhnhq\") pod \"kuadrant-operator-catalog-dnc6t\" (UID: \"29843439-dd60-4557-a0b9-6db34a36a6a9\") " pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:52:49.177328 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.177295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhnhq\" (UniqueName: \"kubernetes.io/projected/29843439-dd60-4557-a0b9-6db34a36a6a9-kube-api-access-rhnhq\") pod \"kuadrant-operator-catalog-dnc6t\" (UID: \"29843439-dd60-4557-a0b9-6db34a36a6a9\") " pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:52:49.187708 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.187673 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhnhq\" (UniqueName: \"kubernetes.io/projected/29843439-dd60-4557-a0b9-6db34a36a6a9-kube-api-access-rhnhq\") pod \"kuadrant-operator-catalog-dnc6t\" (UID: \"29843439-dd60-4557-a0b9-6db34a36a6a9\") " pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:52:49.351115 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.351076 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:52:49.483209 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.483177 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnc6t"] Jul 14 19:52:49.485290 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:52:49.485253 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29843439_dd60_4557_a0b9_6db34a36a6a9.slice/crio-9548a71f99f7a1bbc28e8fd88ddaa95ed17f8c260bf7809867323880b31e3b8b WatchSource:0}: Error finding container 9548a71f99f7a1bbc28e8fd88ddaa95ed17f8c260bf7809867323880b31e3b8b: Status 404 returned error can't find the container with id 9548a71f99f7a1bbc28e8fd88ddaa95ed17f8c260bf7809867323880b31e3b8b Jul 14 19:52:49.779025 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.778935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" event={"ID":"f76c70df-48c4-403f-9d58-0ba6fab78c6b","Type":"ContainerStarted","Data":"c11045d773f3b1502938257f602f3d3ac1243553c7fee909541ccae18996ef13"} Jul 14 19:52:49.780187 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:49.780144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" event={"ID":"29843439-dd60-4557-a0b9-6db34a36a6a9","Type":"ContainerStarted","Data":"9548a71f99f7a1bbc28e8fd88ddaa95ed17f8c260bf7809867323880b31e3b8b"} Jul 14 19:52:51.790703 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:51.790669 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" event={"ID":"f76c70df-48c4-403f-9d58-0ba6fab78c6b","Type":"ContainerStarted","Data":"f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132"} Jul 14 19:52:51.791194 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:51.790720 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" podUID="f76c70df-48c4-403f-9d58-0ba6fab78c6b" containerName="registry-server" containerID="cri-o://f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132" gracePeriod=2 Jul 14 19:52:51.792408 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:51.792378 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" event={"ID":"29843439-dd60-4557-a0b9-6db34a36a6a9","Type":"ContainerStarted","Data":"a4e5f6791983ab82d10ff7ee9414741669e45d21fc0d4fe381dbb5d3af7a2f59"} Jul 14 19:52:51.809461 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:51.809416 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" podStartSLOduration=1.601309129 podStartE2EDuration="3.809401625s" totalImagesPullingTime="2.208092496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:51.807668821 +0000 UTC m=+413.455232385" watchObservedRunningTime="2026-07-14 19:52:51.809401625 +0000 UTC m=+413.456965190" Jul 14 19:52:51.823783 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:51.823739 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" podStartSLOduration=1.164433857 podStartE2EDuration="2.823727332s" totalImagesPullingTime="1.659293475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:52:51.823108334 +0000 UTC m=+413.470671909" watchObservedRunningTime="2026-07-14 19:52:51.823727332 +0000 UTC m=+413.471290896" Jul 14 19:52:52.035079 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.035054 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" Jul 14 19:52:52.103163 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.103127 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpgwb\" (UniqueName: \"kubernetes.io/projected/f76c70df-48c4-403f-9d58-0ba6fab78c6b-kube-api-access-wpgwb\") pod \"f76c70df-48c4-403f-9d58-0ba6fab78c6b\" (UID: \"f76c70df-48c4-403f-9d58-0ba6fab78c6b\") " Jul 14 19:52:52.105313 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.105288 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76c70df-48c4-403f-9d58-0ba6fab78c6b-kube-api-access-wpgwb" (OuterVolumeSpecName: "kube-api-access-wpgwb") pod "f76c70df-48c4-403f-9d58-0ba6fab78c6b" (UID: "f76c70df-48c4-403f-9d58-0ba6fab78c6b"). InnerVolumeSpecName "kube-api-access-wpgwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:52:52.204683 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.204651 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgwb\" (UniqueName: \"kubernetes.io/projected/f76c70df-48c4-403f-9d58-0ba6fab78c6b-kube-api-access-wpgwb\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:52:52.798662 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.798628 2578 generic.go:356] "Generic (PLEG): container finished" podID="f76c70df-48c4-403f-9d58-0ba6fab78c6b" containerID="f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132" exitCode=0 Jul 14 19:52:52.799190 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.798671 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" Jul 14 19:52:52.799190 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.798719 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" event={"ID":"f76c70df-48c4-403f-9d58-0ba6fab78c6b","Type":"ContainerDied","Data":"f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132"} Jul 14 19:52:52.799190 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.798763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kxrdf" event={"ID":"f76c70df-48c4-403f-9d58-0ba6fab78c6b","Type":"ContainerDied","Data":"c11045d773f3b1502938257f602f3d3ac1243553c7fee909541ccae18996ef13"} Jul 14 19:52:52.799190 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.798784 2578 scope.go:117] "RemoveContainer" containerID="f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132" Jul 14 19:52:52.808871 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.808846 2578 scope.go:117] "RemoveContainer" containerID="f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132" Jul 14 19:52:52.809117 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:52:52.809097 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132\": container with ID starting with f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132 not found: ID does not exist" containerID="f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132" Jul 14 19:52:52.809186 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.809125 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132"} err="failed to get container status \"f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132\": rpc error: code = NotFound desc = could not find container \"f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132\": container with ID starting with f7587ddafa34a6599cd3e6ebc83ac4705583c1c57b9ae4c82e170fe95a720132 not found: ID does not exist" Jul 14 19:52:52.821503 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.821482 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kxrdf"] Jul 14 19:52:52.825781 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.825760 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kxrdf"] Jul 14 19:52:52.883414 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:52.883384 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f76c70df-48c4-403f-9d58-0ba6fab78c6b" path="/var/lib/kubelet/pods/f76c70df-48c4-403f-9d58-0ba6fab78c6b/volumes" Jul 14 19:52:59.351772 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:59.351734 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:52:59.352280 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:59.351814 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:52:59.374302 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:59.374271 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:52:59.853070 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:52:59.853043 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-dnc6t" Jul 14 19:53:21.817688 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.817600 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss"] Jul 14 19:53:21.818177 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.817980 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76c70df-48c4-403f-9d58-0ba6fab78c6b" containerName="registry-server" Jul 14 19:53:21.818177 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.817992 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76c70df-48c4-403f-9d58-0ba6fab78c6b" containerName="registry-server" Jul 14 19:53:21.818177 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.818065 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f76c70df-48c4-403f-9d58-0ba6fab78c6b" containerName="registry-server" Jul 14 19:53:21.820885 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.820865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:21.823437 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.823419 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jh6xh\"" Jul 14 19:53:21.832186 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.832165 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss"] Jul 14 19:53:21.954633 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.954598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" (UID: \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:21.954808 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:21.954645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thw2\" (UniqueName: \"kubernetes.io/projected/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-kube-api-access-7thw2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" (UID: \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:22.055121 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:22.055087 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" (UID: \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:22.055279 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:22.055130 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thw2\" (UniqueName: \"kubernetes.io/projected/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-kube-api-access-7thw2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" (UID: \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:22.055465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:22.055441 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" (UID: \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:22.064713 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:22.064681 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thw2\" (UniqueName: \"kubernetes.io/projected/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-kube-api-access-7thw2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" (UID: \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:22.130620 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:22.130540 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:22.271798 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:22.271752 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss"] Jul 14 19:53:22.274823 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:53:22.274794 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a4251b_9c84_47c9_a8b8_c0eaebcd693f.slice/crio-f55b62e365fcd895d24a8d3a6f1c37bcc9052106f02366144fe9cc87a20e49b5 WatchSource:0}: Error finding container f55b62e365fcd895d24a8d3a6f1c37bcc9052106f02366144fe9cc87a20e49b5: Status 404 returned error can't find the container with id f55b62e365fcd895d24a8d3a6f1c37bcc9052106f02366144fe9cc87a20e49b5 Jul 14 19:53:22.940290 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:22.940254 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" event={"ID":"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f","Type":"ContainerStarted","Data":"f55b62e365fcd895d24a8d3a6f1c37bcc9052106f02366144fe9cc87a20e49b5"} Jul 14 19:53:23.740325 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:23.740290 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7"] Jul 14 19:53:23.743805 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:23.743788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" Jul 14 19:53:23.746398 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:23.746369 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 14 19:53:23.746524 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:23.746457 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-4vx5w\"" Jul 14 19:53:23.756067 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:23.756044 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7"] Jul 14 19:53:23.871089 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:23.871053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvbb\" (UniqueName: \"kubernetes.io/projected/13b6802a-e0f2-4e7a-a79b-5994b4f5ca85-kube-api-access-qzvbb\") pod \"dns-operator-controller-manager-864dd78c4c-5qng7\" (UID: \"13b6802a-e0f2-4e7a-a79b-5994b4f5ca85\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" Jul 14 19:53:23.972089 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:23.972044 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvbb\" (UniqueName: \"kubernetes.io/projected/13b6802a-e0f2-4e7a-a79b-5994b4f5ca85-kube-api-access-qzvbb\") pod \"dns-operator-controller-manager-864dd78c4c-5qng7\" (UID: \"13b6802a-e0f2-4e7a-a79b-5994b4f5ca85\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" Jul 14 19:53:23.981980 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:23.981948 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvbb\" (UniqueName: \"kubernetes.io/projected/13b6802a-e0f2-4e7a-a79b-5994b4f5ca85-kube-api-access-qzvbb\") pod \"dns-operator-controller-manager-864dd78c4c-5qng7\" (UID: \"13b6802a-e0f2-4e7a-a79b-5994b4f5ca85\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" Jul 14 19:53:24.055002 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:24.054970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" Jul 14 19:53:24.208987 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:24.208917 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7"] Jul 14 19:53:24.211596 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:53:24.211562 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b6802a_e0f2_4e7a_a79b_5994b4f5ca85.slice/crio-dd6b5472c00d056e14ccfc1d5be2ea1ef17ed0b72e21e6cd553025b21949c668 WatchSource:0}: Error finding container dd6b5472c00d056e14ccfc1d5be2ea1ef17ed0b72e21e6cd553025b21949c668: Status 404 returned error can't find the container with id dd6b5472c00d056e14ccfc1d5be2ea1ef17ed0b72e21e6cd553025b21949c668 Jul 14 19:53:24.953916 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:24.953865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" event={"ID":"13b6802a-e0f2-4e7a-a79b-5994b4f5ca85","Type":"ContainerStarted","Data":"dd6b5472c00d056e14ccfc1d5be2ea1ef17ed0b72e21e6cd553025b21949c668"} Jul 14 19:53:26.967642 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:26.967602 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" event={"ID":"13b6802a-e0f2-4e7a-a79b-5994b4f5ca85","Type":"ContainerStarted","Data":"c5a02d3fcc6d7c9ffa9e3ba6c2c3e68acd674fb7ed91e36da71b13e62bffe6b2"} Jul 14 19:53:26.968073 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:26.967710 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" Jul 14 19:53:26.987677 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:26.987623 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" podStartSLOduration=1.33619095 podStartE2EDuration="3.987609277s" totalImagesPullingTime="2.651418327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:26.985192502 +0000 UTC m=+448.632756069" watchObservedRunningTime="2026-07-14 19:53:26.987609277 +0000 UTC m=+448.635172910" Jul 14 19:53:28.982762 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:28.982719 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" event={"ID":"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f","Type":"ContainerStarted","Data":"8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab"} Jul 14 19:53:28.983292 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:28.982876 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:29.005023 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:29.004971 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" podStartSLOduration=1.9488455980000001 podStartE2EDuration="8.00495549s" totalImagesPullingTime="6.056109892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:29.002322556 +0000 UTC m=+450.649886146" watchObservedRunningTime="2026-07-14 19:53:29.00495549 +0000 UTC m=+450.652519055" Jul 14 19:53:30.025576 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.025542 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf"] Jul 14 19:53:30.029061 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.029044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" Jul 14 19:53:30.032092 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.032075 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4h4g5\"" Jul 14 19:53:30.039717 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.039693 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf"] Jul 14 19:53:30.131231 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.131195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tn2x\" (UniqueName: \"kubernetes.io/projected/a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f-kube-api-access-4tn2x\") pod \"authorino-operator-75cc8d5cb7-ndrxf\" (UID: \"a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" Jul 14 19:53:30.232618 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.232581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tn2x\" (UniqueName: \"kubernetes.io/projected/a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f-kube-api-access-4tn2x\") pod \"authorino-operator-75cc8d5cb7-ndrxf\" (UID: \"a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" Jul 14 19:53:30.245184 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.245153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tn2x\" (UniqueName: \"kubernetes.io/projected/a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f-kube-api-access-4tn2x\") pod \"authorino-operator-75cc8d5cb7-ndrxf\" (UID: \"a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" Jul 14 19:53:30.341131 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.341091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" Jul 14 19:53:30.481519 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.481495 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf"] Jul 14 19:53:30.483447 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:53:30.483417 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a6291d_fe14_4f8f_838d_9dc5c7b19f7f.slice/crio-ccbdd112722b7d3d3c6c3e9efa62f983f2d5fda33ce31e5cfab785e69340a3d6 WatchSource:0}: Error finding container ccbdd112722b7d3d3c6c3e9efa62f983f2d5fda33ce31e5cfab785e69340a3d6: Status 404 returned error can't find the container with id ccbdd112722b7d3d3c6c3e9efa62f983f2d5fda33ce31e5cfab785e69340a3d6 Jul 14 19:53:30.993870 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:30.993837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" event={"ID":"a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f","Type":"ContainerStarted","Data":"ccbdd112722b7d3d3c6c3e9efa62f983f2d5fda33ce31e5cfab785e69340a3d6"} Jul 14 19:53:31.376854 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.376795 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49"] Jul 14 19:53:31.381485 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.381465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.384246 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.384221 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 14 19:53:31.384381 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.384227 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 14 19:53:31.384381 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.384227 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-slfsg\"" Jul 14 19:53:31.389946 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.389920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49"] Jul 14 19:53:31.546193 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.546163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5n7b\" (UniqueName: \"kubernetes.io/projected/a56f7b6f-6794-457e-aae9-9422e3e8c243-kube-api-access-n5n7b\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.546361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.546210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a56f7b6f-6794-457e-aae9-9422e3e8c243-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.546361 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.546237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a56f7b6f-6794-457e-aae9-9422e3e8c243-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.647333 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.647231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a56f7b6f-6794-457e-aae9-9422e3e8c243-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.647333 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.647290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a56f7b6f-6794-457e-aae9-9422e3e8c243-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.647566 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.647374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5n7b\" (UniqueName: \"kubernetes.io/projected/a56f7b6f-6794-457e-aae9-9422e3e8c243-kube-api-access-n5n7b\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.647896 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.647863 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a56f7b6f-6794-457e-aae9-9422e3e8c243-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.649721 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.649699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a56f7b6f-6794-457e-aae9-9422e3e8c243-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.657204 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.657183 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5n7b\" (UniqueName: \"kubernetes.io/projected/a56f7b6f-6794-457e-aae9-9422e3e8c243-kube-api-access-n5n7b\") pod \"kuadrant-console-plugin-66b5774985-pkl49\" (UID: \"a56f7b6f-6794-457e-aae9-9422e3e8c243\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.692560 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.692528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" Jul 14 19:53:31.853541 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:31.853494 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49"] Jul 14 19:53:31.856867 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:53:31.856798 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda56f7b6f_6794_457e_aae9_9422e3e8c243.slice/crio-2495f2ce55227d212ffca0ad552532b88ce9018c0930b0c59a1cfc6d304640a0 WatchSource:0}: Error finding container 2495f2ce55227d212ffca0ad552532b88ce9018c0930b0c59a1cfc6d304640a0: Status 404 returned error can't find the container with id 2495f2ce55227d212ffca0ad552532b88ce9018c0930b0c59a1cfc6d304640a0 Jul 14 19:53:32.001495 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:32.001394 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" event={"ID":"a56f7b6f-6794-457e-aae9-9422e3e8c243","Type":"ContainerStarted","Data":"2495f2ce55227d212ffca0ad552532b88ce9018c0930b0c59a1cfc6d304640a0"} Jul 14 19:53:34.015815 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:34.015777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" event={"ID":"a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f","Type":"ContainerStarted","Data":"459e4364d5da9226bc06cc926783742edf1494eb0f1a87180be505ce1154a27b"} Jul 14 19:53:34.016281 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:34.015906 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" Jul 14 19:53:34.034221 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:34.034165 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" podStartSLOduration=1.179874053 podStartE2EDuration="4.034144678s" totalImagesPullingTime="2.854270625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:34.032795024 +0000 UTC m=+455.680358591" watchObservedRunningTime="2026-07-14 19:53:34.034144678 +0000 UTC m=+455.681708244" Jul 14 19:53:37.975768 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:37.975613 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5qng7" Jul 14 19:53:39.989940 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:39.989900 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:41.707814 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.707770 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss"] Jul 14 19:53:41.708304 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.708014 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" containerName="manager" containerID="cri-o://8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab" gracePeriod=2 Jul 14 19:53:41.715746 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.715715 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss"] Jul 14 19:53:41.719266 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.719223 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86"] Jul 14 19:53:41.719811 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.719789 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" containerName="manager" Jul 14 19:53:41.719989 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.719963 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" containerName="manager" Jul 14 19:53:41.720390 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.720371 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" containerName="manager" Jul 14 19:53:41.727262 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.727238 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:41.739224 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.739187 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86"] Jul 14 19:53:41.748663 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.748631 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv"] Jul 14 19:53:41.752874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.752820 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:41.788633 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.788606 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv"] Jul 14 19:53:41.803946 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:53:41.803901 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" is forbidden: User \"system:node:ip-10-0-133-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-243.ec2.internal' and this object" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:41.814182 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:53:41.814143 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" is forbidden: User \"system:node:ip-10-0-133-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-243.ec2.internal' and this object" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:41.846057 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.846017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mzl9\" (UniqueName: \"kubernetes.io/projected/6114fb94-9b0f-4898-b470-55f87fdf371f-kube-api-access-7mzl9\") pod \"kuadrant-operator-controller-manager-86878c769f-5hm86\" (UID: \"6114fb94-9b0f-4898-b470-55f87fdf371f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:41.846258 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.846158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjmsb\" (UniqueName: \"kubernetes.io/projected/77b9746e-68a7-4c8c-844e-5c242dae938e-kube-api-access-zjmsb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mmphv\" (UID: \"77b9746e-68a7-4c8c-844e-5c242dae938e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:41.846258 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.846235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77b9746e-68a7-4c8c-844e-5c242dae938e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mmphv\" (UID: \"77b9746e-68a7-4c8c-844e-5c242dae938e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:41.846385 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.846343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6114fb94-9b0f-4898-b470-55f87fdf371f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5hm86\" (UID: \"6114fb94-9b0f-4898-b470-55f87fdf371f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:41.947014 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.946970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6114fb94-9b0f-4898-b470-55f87fdf371f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5hm86\" (UID: \"6114fb94-9b0f-4898-b470-55f87fdf371f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:41.947205 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.947058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mzl9\" (UniqueName: \"kubernetes.io/projected/6114fb94-9b0f-4898-b470-55f87fdf371f-kube-api-access-7mzl9\") pod \"kuadrant-operator-controller-manager-86878c769f-5hm86\" (UID: \"6114fb94-9b0f-4898-b470-55f87fdf371f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:41.947205 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.947140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjmsb\" (UniqueName: \"kubernetes.io/projected/77b9746e-68a7-4c8c-844e-5c242dae938e-kube-api-access-zjmsb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mmphv\" (UID: \"77b9746e-68a7-4c8c-844e-5c242dae938e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:41.947205 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.947189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77b9746e-68a7-4c8c-844e-5c242dae938e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mmphv\" (UID: \"77b9746e-68a7-4c8c-844e-5c242dae938e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:41.947472 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.947442 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6114fb94-9b0f-4898-b470-55f87fdf371f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5hm86\" (UID: \"6114fb94-9b0f-4898-b470-55f87fdf371f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:41.947620 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.947597 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77b9746e-68a7-4c8c-844e-5c242dae938e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mmphv\" (UID: \"77b9746e-68a7-4c8c-844e-5c242dae938e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:41.959886 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.959811 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mzl9\" (UniqueName: \"kubernetes.io/projected/6114fb94-9b0f-4898-b470-55f87fdf371f-kube-api-access-7mzl9\") pod \"kuadrant-operator-controller-manager-86878c769f-5hm86\" (UID: \"6114fb94-9b0f-4898-b470-55f87fdf371f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:41.959886 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:41.959814 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjmsb\" (UniqueName: \"kubernetes.io/projected/77b9746e-68a7-4c8c-844e-5c242dae938e-kube-api-access-zjmsb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mmphv\" (UID: \"77b9746e-68a7-4c8c-844e-5c242dae938e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:42.083730 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:42.083691 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:42.092918 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:42.092889 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:45.022967 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:45.022930 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ndrxf" Jul 14 19:53:55.602966 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.602928 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv"] Jul 14 19:53:55.606707 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:53:55.606664 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b9746e_68a7_4c8c_844e_5c242dae938e.slice/crio-2b802cf7c6e6162c2e1679fc549b005b0b0a51fd4b4bd094f089fafdaf61f418 WatchSource:0}: Error finding container 2b802cf7c6e6162c2e1679fc549b005b0b0a51fd4b4bd094f089fafdaf61f418: Status 404 returned error can't find the container with id 2b802cf7c6e6162c2e1679fc549b005b0b0a51fd4b4bd094f089fafdaf61f418 Jul 14 19:53:55.607886 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.607857 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:55.610249 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:53:55.610217 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" is forbidden: User \"system:node:ip-10-0-133-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-243.ec2.internal' and this object" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:55.628565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.628538 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86"] Jul 14 19:53:55.638996 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:53:55.638953 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6114fb94_9b0f_4898_b470_55f87fdf371f.slice/crio-9471dff01a5b3fc88bf38243d4d185a9c8ed3119bf4805c6b1f77ffe007db6dd WatchSource:0}: Error finding container 9471dff01a5b3fc88bf38243d4d185a9c8ed3119bf4805c6b1f77ffe007db6dd: Status 404 returned error can't find the container with id 9471dff01a5b3fc88bf38243d4d185a9c8ed3119bf4805c6b1f77ffe007db6dd Jul 14 19:53:55.674715 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.674695 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7thw2\" (UniqueName: \"kubernetes.io/projected/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-kube-api-access-7thw2\") pod \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\" (UID: \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\") " Jul 14 19:53:55.674846 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.674750 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-extensions-socket-volume\") pod \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\" (UID: \"f3a4251b-9c84-47c9-a8b8-c0eaebcd693f\") " Jul 14 19:53:55.675273 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.675240 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" (UID: "f3a4251b-9c84-47c9-a8b8-c0eaebcd693f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:53:55.676935 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.676906 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-kube-api-access-7thw2" (OuterVolumeSpecName: "kube-api-access-7thw2") pod "f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" (UID: "f3a4251b-9c84-47c9-a8b8-c0eaebcd693f"). InnerVolumeSpecName "kube-api-access-7thw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:53:55.775536 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.775497 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7thw2\" (UniqueName: \"kubernetes.io/projected/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-kube-api-access-7thw2\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:53:55.775536 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:55.775529 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f-extensions-socket-volume\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:53:56.147230 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.147185 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" event={"ID":"a56f7b6f-6794-457e-aae9-9422e3e8c243","Type":"ContainerStarted","Data":"7ddaa7859b16e9c635d9ad9357b39ea5c5b480ca739b6e11c227a6a433a10de9"} Jul 14 19:53:56.148902 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.148873 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" event={"ID":"77b9746e-68a7-4c8c-844e-5c242dae938e","Type":"ContainerStarted","Data":"5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e"} Jul 14 19:53:56.149093 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.148925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" event={"ID":"77b9746e-68a7-4c8c-844e-5c242dae938e","Type":"ContainerStarted","Data":"2b802cf7c6e6162c2e1679fc549b005b0b0a51fd4b4bd094f089fafdaf61f418"} Jul 14 19:53:56.149093 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.148958 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:53:56.150163 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.150144 2578 generic.go:356] "Generic (PLEG): container finished" podID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" containerID="8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab" exitCode=0 Jul 14 19:53:56.150219 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.150179 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:56.150259 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.150241 2578 scope.go:117] "RemoveContainer" containerID="8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab" Jul 14 19:53:56.151882 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.151861 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" event={"ID":"6114fb94-9b0f-4898-b470-55f87fdf371f","Type":"ContainerStarted","Data":"bbc4dd655e82d14ad81e7537b04a1126976538c53ae08804c0b86744c4913740"} Jul 14 19:53:56.151974 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.151899 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" event={"ID":"6114fb94-9b0f-4898-b470-55f87fdf371f","Type":"ContainerStarted","Data":"9471dff01a5b3fc88bf38243d4d185a9c8ed3119bf4805c6b1f77ffe007db6dd"} Jul 14 19:53:56.151974 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.151953 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:53:56.159642 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.159627 2578 scope.go:117] "RemoveContainer" containerID="8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab" Jul 14 19:53:56.159942 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:53:56.159919 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab\": container with ID starting with 8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab not found: ID does not exist" containerID="8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab" Jul 14 19:53:56.160016 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.159953 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab"} err="failed to get container status \"8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab\": rpc error: code = NotFound desc = could not find container \"8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab\": container with ID starting with 8bcde3f7305498ec6de585c79dce359020dc1413dd703ac9602d9dfb6f4db3ab not found: ID does not exist" Jul 14 19:53:56.169445 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.169395 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pkl49" podStartSLOduration=1.491895899 podStartE2EDuration="25.169382821s" totalImagesPullingTime="23.677486922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:56.168298277 +0000 UTC m=+477.815861842" watchObservedRunningTime="2026-07-14 19:53:56.169382821 +0000 UTC m=+477.816946388" Jul 14 19:53:56.170202 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:53:56.170178 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" is forbidden: User \"system:node:ip-10-0-133-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-243.ec2.internal' and this object" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:56.171991 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:53:56.171972 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-m2qss\" is forbidden: User \"system:node:ip-10-0-133-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-243.ec2.internal' and this object" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m2qss" Jul 14 19:53:56.196700 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.196647 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" podStartSLOduration=15.196630273 podStartE2EDuration="15.196630273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:56.195638923 +0000 UTC m=+477.843202510" watchObservedRunningTime="2026-07-14 19:53:56.196630273 +0000 UTC m=+477.844193839" Jul 14 19:53:56.218579 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.218530 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" podStartSLOduration=15.218515626 podStartE2EDuration="15.218515626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:53:56.217972074 +0000 UTC m=+477.865535652" watchObservedRunningTime="2026-07-14 19:53:56.218515626 +0000 UTC m=+477.866079191" Jul 14 19:53:56.884352 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:56.884315 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a4251b-9c84-47c9-a8b8-c0eaebcd693f" path="/var/lib/kubelet/pods/f3a4251b-9c84-47c9-a8b8-c0eaebcd693f/volumes" Jul 14 19:53:58.812785 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:53:58.812751 2578 scope.go:117] "RemoveContainer" containerID="93d1196253104bc645a59d55a085637aad4bc15c82a59a2e4444cde85b5f695e" Jul 14 19:54:07.159992 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.159954 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 19:54:07.160470 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.160042 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:54:07.261291 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.261255 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv"] Jul 14 19:54:07.261483 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.261459 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" podUID="77b9746e-68a7-4c8c-844e-5c242dae938e" containerName="manager" containerID="cri-o://5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e" gracePeriod=10 Jul 14 19:54:07.523010 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.522985 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:54:07.684431 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.684334 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjmsb\" (UniqueName: \"kubernetes.io/projected/77b9746e-68a7-4c8c-844e-5c242dae938e-kube-api-access-zjmsb\") pod \"77b9746e-68a7-4c8c-844e-5c242dae938e\" (UID: \"77b9746e-68a7-4c8c-844e-5c242dae938e\") " Jul 14 19:54:07.684584 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.684445 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77b9746e-68a7-4c8c-844e-5c242dae938e-extensions-socket-volume\") pod \"77b9746e-68a7-4c8c-844e-5c242dae938e\" (UID: \"77b9746e-68a7-4c8c-844e-5c242dae938e\") " Jul 14 19:54:07.684874 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.684816 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b9746e-68a7-4c8c-844e-5c242dae938e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "77b9746e-68a7-4c8c-844e-5c242dae938e" (UID: "77b9746e-68a7-4c8c-844e-5c242dae938e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:54:07.686456 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.686435 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b9746e-68a7-4c8c-844e-5c242dae938e-kube-api-access-zjmsb" (OuterVolumeSpecName: "kube-api-access-zjmsb") pod "77b9746e-68a7-4c8c-844e-5c242dae938e" (UID: "77b9746e-68a7-4c8c-844e-5c242dae938e"). InnerVolumeSpecName "kube-api-access-zjmsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:54:07.785444 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.785404 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/77b9746e-68a7-4c8c-844e-5c242dae938e-extensions-socket-volume\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:54:07.785444 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:07.785438 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjmsb\" (UniqueName: \"kubernetes.io/projected/77b9746e-68a7-4c8c-844e-5c242dae938e-kube-api-access-zjmsb\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 19:54:08.217873 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.217840 2578 generic.go:356] "Generic (PLEG): container finished" podID="77b9746e-68a7-4c8c-844e-5c242dae938e" containerID="5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e" exitCode=0 Jul 14 19:54:08.218269 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.217896 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" Jul 14 19:54:08.218269 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.217926 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" event={"ID":"77b9746e-68a7-4c8c-844e-5c242dae938e","Type":"ContainerDied","Data":"5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e"} Jul 14 19:54:08.218269 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.217980 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv" event={"ID":"77b9746e-68a7-4c8c-844e-5c242dae938e","Type":"ContainerDied","Data":"2b802cf7c6e6162c2e1679fc549b005b0b0a51fd4b4bd094f089fafdaf61f418"} Jul 14 19:54:08.218269 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.218001 2578 scope.go:117] "RemoveContainer" containerID="5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e" Jul 14 19:54:08.226942 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.226921 2578 scope.go:117] "RemoveContainer" containerID="5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e" Jul 14 19:54:08.227232 ip-10-0-133-243 kubenswrapper[2578]: E0714 19:54:08.227211 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e\": container with ID starting with 5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e not found: ID does not exist" containerID="5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e" Jul 14 19:54:08.227283 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.227243 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e"} err="failed to get container status \"5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e\": rpc error: code = NotFound desc = could not find container \"5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e\": container with ID starting with 5013acad717c979b71623fa862e1b8a48183b335370334ab4856acf74e05131e not found: ID does not exist" Jul 14 19:54:08.246046 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.246016 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv"] Jul 14 19:54:08.252371 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.252345 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mmphv"] Jul 14 19:54:08.884021 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:08.883974 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77b9746e-68a7-4c8c-844e-5c242dae938e" path="/var/lib/kubelet/pods/77b9746e-68a7-4c8c-844e-5c242dae938e/volumes" Jul 14 19:54:23.713478 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:23.713441 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr"] Jul 14 19:54:23.714048 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:23.714025 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b9746e-68a7-4c8c-844e-5c242dae938e" containerName="manager" Jul 14 19:54:23.714048 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:23.714041 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b9746e-68a7-4c8c-844e-5c242dae938e" containerName="manager" Jul 14 19:54:23.714165 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:23.714140 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b9746e-68a7-4c8c-844e-5c242dae938e" containerName="manager" Jul 14 19:54:23.994930 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:23.994837 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr"] Jul 14 19:54:23.994930 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:23.994927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:23.997645 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:23.997622 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zfbzn\"" Jul 14 19:54:24.125947 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.125904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.126143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.125960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm96x\" (UniqueName: \"kubernetes.io/projected/a42a78ef-b193-4fc1-8196-2b1979b4a50f-kube-api-access-mm96x\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.126143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.125984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.126143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.126003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.126143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.126028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.126143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.126061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.126143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.126099 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.126143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.126114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.126143 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.126147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227204 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227204 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm96x\" (UniqueName: \"kubernetes.io/projected/a42a78ef-b193-4fc1-8196-2b1979b4a50f-kube-api-access-mm96x\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227482 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227482 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227482 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227482 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227482 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227482 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227482 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227801 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227764 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227881 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227938 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227880 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.227976 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.227940 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.228406 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.228371 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.229898 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.229879 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.229973 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.229935 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.238559 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.238533 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a42a78ef-b193-4fc1-8196-2b1979b4a50f-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.238672 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.238633 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm96x\" (UniqueName: \"kubernetes.io/projected/a42a78ef-b193-4fc1-8196-2b1979b4a50f-kube-api-access-mm96x\") pod \"maas-default-gateway-openshift-default-56654dcf7-pthzr\" (UID: \"a42a78ef-b193-4fc1-8196-2b1979b4a50f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.305502 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.305470 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:24.505860 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.505818 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr"] Jul 14 19:54:24.508062 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:54:24.508028 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42a78ef_b193_4fc1_8196_2b1979b4a50f.slice/crio-95234633220bf8e432f19128e27f010a573012dbd5f8283ae0c22395a9765088 WatchSource:0}: Error finding container 95234633220bf8e432f19128e27f010a573012dbd5f8283ae0c22395a9765088: Status 404 returned error can't find the container with id 95234633220bf8e432f19128e27f010a573012dbd5f8283ae0c22395a9765088 Jul 14 19:54:24.510301 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.510267 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:54:24.510392 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.510337 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:54:24.510392 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:24.510367 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:54:25.305090 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:25.305052 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" event={"ID":"a42a78ef-b193-4fc1-8196-2b1979b4a50f","Type":"ContainerStarted","Data":"08c560ac51266448f25385cc3bc83056ba7fbe6ba912a88f6d15985397334d50"} Jul 14 19:54:25.305528 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:25.305108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" event={"ID":"a42a78ef-b193-4fc1-8196-2b1979b4a50f","Type":"ContainerStarted","Data":"95234633220bf8e432f19128e27f010a573012dbd5f8283ae0c22395a9765088"} Jul 14 19:54:25.306104 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:25.306085 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:25.311223 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:25.311199 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:25.329451 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:25.329400 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" podStartSLOduration=2.329384716 podStartE2EDuration="2.329384716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:54:25.328004703 +0000 UTC m=+506.975568272" watchObservedRunningTime="2026-07-14 19:54:25.329384716 +0000 UTC m=+506.976948281" Jul 14 19:54:26.310402 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:26.310368 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:54:26.311393 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:54:26.311376 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:55:43.697650 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.697614 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-flfq6"] Jul 14 19:55:43.701381 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.701357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:43.703906 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.703876 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 19:55:43.713229 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.713196 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-flfq6"] Jul 14 19:55:43.727776 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.727741 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-vjgzw"] Jul 14 19:55:43.731365 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.731346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:43.733884 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.733867 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g8cqc\"" Jul 14 19:55:43.742578 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.742556 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-vjgzw"] Jul 14 19:55:43.799042 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.799007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456b87f2-0077-4159-8049-4eae24c13fd9-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-vjgzw\" (UID: \"456b87f2-0077-4159-8049-4eae24c13fd9\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:43.799216 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.799052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxbt\" (UniqueName: \"kubernetes.io/projected/456b87f2-0077-4159-8049-4eae24c13fd9-kube-api-access-wsxbt\") pod \"payload-processing-f9b8d9b9b-vjgzw\" (UID: \"456b87f2-0077-4159-8049-4eae24c13fd9\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:43.799216 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.799150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvl6\" (UniqueName: \"kubernetes.io/projected/a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4-kube-api-access-dzvl6\") pod \"payload-pre-processing-74584bb9f8-flfq6\" (UID: \"a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:43.799216 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.799207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-flfq6\" (UID: \"a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:43.900565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.900528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456b87f2-0077-4159-8049-4eae24c13fd9-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-vjgzw\" (UID: \"456b87f2-0077-4159-8049-4eae24c13fd9\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:43.900764 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.900594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxbt\" (UniqueName: \"kubernetes.io/projected/456b87f2-0077-4159-8049-4eae24c13fd9-kube-api-access-wsxbt\") pod \"payload-processing-f9b8d9b9b-vjgzw\" (UID: \"456b87f2-0077-4159-8049-4eae24c13fd9\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:43.900764 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.900681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvl6\" (UniqueName: \"kubernetes.io/projected/a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4-kube-api-access-dzvl6\") pod \"payload-pre-processing-74584bb9f8-flfq6\" (UID: \"a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:43.900764 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.900730 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-flfq6\" (UID: \"a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:43.901476 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.901450 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-flfq6\" (UID: \"a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:43.901572 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.901510 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/456b87f2-0077-4159-8049-4eae24c13fd9-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-vjgzw\" (UID: \"456b87f2-0077-4159-8049-4eae24c13fd9\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:43.911280 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.911258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxbt\" (UniqueName: \"kubernetes.io/projected/456b87f2-0077-4159-8049-4eae24c13fd9-kube-api-access-wsxbt\") pod \"payload-processing-f9b8d9b9b-vjgzw\" (UID: \"456b87f2-0077-4159-8049-4eae24c13fd9\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:43.911365 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:43.911335 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvl6\" (UniqueName: \"kubernetes.io/projected/a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4-kube-api-access-dzvl6\") pod \"payload-pre-processing-74584bb9f8-flfq6\" (UID: \"a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:44.011988 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:44.011902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:44.042083 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:44.042047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:44.167948 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:44.167915 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-flfq6"] Jul 14 19:55:44.170195 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:55:44.170156 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cbaa00_d31c_4c98_94bd_ba07e8d1f9b4.slice/crio-f22c8be1120d28be158aa586d8f4b2a0a88a4f39e08af2266fecb08764ac283d WatchSource:0}: Error finding container f22c8be1120d28be158aa586d8f4b2a0a88a4f39e08af2266fecb08764ac283d: Status 404 returned error can't find the container with id f22c8be1120d28be158aa586d8f4b2a0a88a4f39e08af2266fecb08764ac283d Jul 14 19:55:44.405801 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:44.405776 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-vjgzw"] Jul 14 19:55:44.407682 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:55:44.407657 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456b87f2_0077_4159_8049_4eae24c13fd9.slice/crio-c1344137338361689aa9c6514b59148aef1ec3272b7c618399d43ce08d5d8723 WatchSource:0}: Error finding container c1344137338361689aa9c6514b59148aef1ec3272b7c618399d43ce08d5d8723: Status 404 returned error can't find the container with id c1344137338361689aa9c6514b59148aef1ec3272b7c618399d43ce08d5d8723 Jul 14 19:55:44.703929 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:44.703783 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" event={"ID":"456b87f2-0077-4159-8049-4eae24c13fd9","Type":"ContainerStarted","Data":"c1344137338361689aa9c6514b59148aef1ec3272b7c618399d43ce08d5d8723"} Jul 14 19:55:44.705246 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:44.705216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" event={"ID":"a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4","Type":"ContainerStarted","Data":"f22c8be1120d28be158aa586d8f4b2a0a88a4f39e08af2266fecb08764ac283d"} Jul 14 19:55:46.718303 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:46.718268 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" event={"ID":"456b87f2-0077-4159-8049-4eae24c13fd9","Type":"ContainerStarted","Data":"b5a670704f04e21ff7e7e215e19d08684296232ac483ebc0aca19de4f59c3e20"} Jul 14 19:55:46.718773 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:46.718369 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:46.719872 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:46.719848 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" event={"ID":"a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4","Type":"ContainerStarted","Data":"c04952798ab8de22e6e0a4ef328d907a7277527a4b190a54bc269d2966305eac"} Jul 14 19:55:46.719951 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:46.719897 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:46.736634 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:46.736558 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" podStartSLOduration=2.087349771 podStartE2EDuration="3.736539509s" totalImagesPullingTime="1.649189738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:46.734406102 +0000 UTC m=+588.381969669" watchObservedRunningTime="2026-07-14 19:55:46.736539509 +0000 UTC m=+588.384103060" Jul 14 19:55:46.754869 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:46.754789 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" podStartSLOduration=1.868320429 podStartE2EDuration="3.754767578s" totalImagesPullingTime="1.886447149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:55:46.750792952 +0000 UTC m=+588.398356519" watchObservedRunningTime="2026-07-14 19:55:46.754767578 +0000 UTC m=+588.402331140" Jul 14 19:55:51.747232 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:51.747149 2578 generic.go:356] "Generic (PLEG): container finished" podID="a42a78ef-b193-4fc1-8196-2b1979b4a50f" containerID="08c560ac51266448f25385cc3bc83056ba7fbe6ba912a88f6d15985397334d50" exitCode=0 Jul 14 19:55:51.747626 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:51.747226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" event={"ID":"a42a78ef-b193-4fc1-8196-2b1979b4a50f","Type":"ContainerDied","Data":"08c560ac51266448f25385cc3bc83056ba7fbe6ba912a88f6d15985397334d50"} Jul 14 19:55:51.747696 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:51.747680 2578 scope.go:117] "RemoveContainer" containerID="08c560ac51266448f25385cc3bc83056ba7fbe6ba912a88f6d15985397334d50" Jul 14 19:55:51.748813 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:51.748787 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:55:51.748896 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:51.748876 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:55:51.748933 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:51.748908 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 19:55:52.758060 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:52.758016 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" event={"ID":"a42a78ef-b193-4fc1-8196-2b1979b4a50f","Type":"ContainerStarted","Data":"07218004b23f551c0c92571caa9592bd984e035d201a52a3ecd9a6efe24c808b"} Jul 14 19:55:53.306224 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:53.306179 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:55:53.307489 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:53.307463 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pthzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 14 19:55:53.307552 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:53.307514 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" podUID="a42a78ef-b193-4fc1-8196-2b1979b4a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 14 19:55:54.306736 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:54.306695 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pthzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 14 19:55:54.307163 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:54.306750 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" podUID="a42a78ef-b193-4fc1-8196-2b1979b4a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 14 19:55:55.306233 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:55.306194 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pthzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 14 19:55:55.306424 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:55.306285 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" podUID="a42a78ef-b193-4fc1-8196-2b1979b4a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 14 19:55:56.306072 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:56.306037 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pthzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 14 19:55:56.306475 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:56.306091 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" podUID="a42a78ef-b193-4fc1-8196-2b1979b4a50f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 14 19:55:56.311194 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:56.311166 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:55:57.309565 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:57.309532 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:55:57.310532 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:57.310516 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pthzr" Jul 14 19:55:57.727456 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:57.727366 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-vjgzw" Jul 14 19:55:57.727456 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:57.727448 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-flfq6" Jul 14 19:55:58.808531 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:58.808499 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 19:55:58.809040 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:58.808738 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 19:55:58.814900 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:58.814878 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:55:58.815034 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:55:58.815016 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 19:56:29.594778 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.594685 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-782qh"] Jul 14 19:56:29.597465 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.597443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.600108 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.600078 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 19:56:29.601157 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.601129 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wjq24\"" Jul 14 19:56:29.601274 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.601158 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 19:56:29.601274 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.601219 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 14 19:56:29.607388 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.607362 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-782qh"] Jul 14 19:56:29.623908 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.623872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xmz\" (UniqueName: \"kubernetes.io/projected/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-kube-api-access-l5xmz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.624157 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.623958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.624157 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.624091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.624157 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.624135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.624360 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.624172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.624360 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.624223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.724914 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.724858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.724914 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.724917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.725176 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.724950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.725176 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.724974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.725176 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.725004 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xmz\" (UniqueName: \"kubernetes.io/projected/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-kube-api-access-l5xmz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.725176 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.725068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.725375 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.725348 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.725453 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.725411 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.725544 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.725522 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.727262 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.727231 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.727513 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.727494 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.737200 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.737159 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xmz\" (UniqueName: \"kubernetes.io/projected/9a8a8afd-9c65-46b4-931c-94e0c774b2d7-kube-api-access-l5xmz\") pod \"e2e-trlp-test-simulated-kserve-598466548d-782qh\" (UID: \"9a8a8afd-9c65-46b4-931c-94e0c774b2d7\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:29.910814 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:29.910714 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:30.050216 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:30.050181 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-782qh"] Jul 14 19:56:30.054121 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:56:30.054092 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8a8afd_9c65_46b4_931c_94e0c774b2d7.slice/crio-a9fb4ea03ed610327ef535576210c78e189301de6b0285191ea41b8bf2233037 WatchSource:0}: Error finding container a9fb4ea03ed610327ef535576210c78e189301de6b0285191ea41b8bf2233037: Status 404 returned error can't find the container with id a9fb4ea03ed610327ef535576210c78e189301de6b0285191ea41b8bf2233037 Jul 14 19:56:30.055438 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:30.055422 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:56:30.949460 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:30.949411 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" event={"ID":"9a8a8afd-9c65-46b4-931c-94e0c774b2d7","Type":"ContainerStarted","Data":"a9fb4ea03ed610327ef535576210c78e189301de6b0285191ea41b8bf2233037"} Jul 14 19:56:31.956155 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:31.956113 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" event={"ID":"9a8a8afd-9c65-46b4-931c-94e0c774b2d7","Type":"ContainerStarted","Data":"90423fa6861cf7100cc7db09cbca1d7a060cebe4f30a832e40148bd8b6b9a329"} Jul 14 19:56:31.991596 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:31.991544 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" podStartSLOduration=1.9906923669999999 podStartE2EDuration="2.991526608s" totalImagesPullingTime="1.000834241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:31.989227816 +0000 UTC m=+633.636791384" watchObservedRunningTime="2026-07-14 19:56:31.991526608 +0000 UTC m=+633.639090173" Jul 14 19:56:39.911285 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:39.911235 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:39.911779 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:39.911303 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:39.924149 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:39.924125 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:40.011499 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:40.011464 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-782qh" Jul 14 19:56:53.906061 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:53.906026 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2"] Jul 14 19:56:53.909534 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:53.909512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:53.912222 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:53.912202 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 14 19:56:53.920736 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:53.920711 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2"] Jul 14 19:56:54.053340 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.053288 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21118a9c-ef38-4929-ae38-bf1e14e0be71-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.053513 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.053420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.053513 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.053497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.053596 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.053521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.053596 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.053577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhj6\" (UniqueName: \"kubernetes.io/projected/21118a9c-ef38-4929-ae38-bf1e14e0be71-kube-api-access-9nhj6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.053675 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.053622 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155099 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155299 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155299 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155168 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155299 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhj6\" (UniqueName: \"kubernetes.io/projected/21118a9c-ef38-4929-ae38-bf1e14e0be71-kube-api-access-9nhj6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155299 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155299 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21118a9c-ef38-4929-ae38-bf1e14e0be71-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155601 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155571 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155667 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.155716 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.155668 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.157433 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.157373 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21118a9c-ef38-4929-ae38-bf1e14e0be71-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.157666 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.157646 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21118a9c-ef38-4929-ae38-bf1e14e0be71-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.164608 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.164577 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhj6\" (UniqueName: \"kubernetes.io/projected/21118a9c-ef38-4929-ae38-bf1e14e0be71-kube-api-access-9nhj6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-58mq2\" (UID: \"21118a9c-ef38-4929-ae38-bf1e14e0be71\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.220903 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.220865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:56:54.363639 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:54.363607 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2"] Jul 14 19:56:54.365443 ip-10-0-133-243 kubenswrapper[2578]: W0714 19:56:54.365411 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21118a9c_ef38_4929_ae38_bf1e14e0be71.slice/crio-9600444aeb09c13eff205e617bf26b596f42be0c85f18125e8b9db31cd7e4f94 WatchSource:0}: Error finding container 9600444aeb09c13eff205e617bf26b596f42be0c85f18125e8b9db31cd7e4f94: Status 404 returned error can't find the container with id 9600444aeb09c13eff205e617bf26b596f42be0c85f18125e8b9db31cd7e4f94 Jul 14 19:56:55.074269 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:55.074226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" event={"ID":"21118a9c-ef38-4929-ae38-bf1e14e0be71","Type":"ContainerStarted","Data":"8a7322a228725772bbd52696304ef97ef31bdff6874ac5e17f557d1519f8a212"} Jul 14 19:56:55.074767 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:55.074278 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" event={"ID":"21118a9c-ef38-4929-ae38-bf1e14e0be71","Type":"ContainerStarted","Data":"9600444aeb09c13eff205e617bf26b596f42be0c85f18125e8b9db31cd7e4f94"} Jul 14 19:56:55.092396 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:56:55.092330 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" podStartSLOduration=1.893424003 podStartE2EDuration="2.092316337s" totalImagesPullingTime="198.892334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:56:55.091852704 +0000 UTC m=+656.739416266" watchObservedRunningTime="2026-07-14 19:56:55.092316337 +0000 UTC m=+656.739879910" Jul 14 19:57:04.221464 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:57:04.221415 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:57:04.222084 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:57:04.221491 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:57:04.234518 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:57:04.234486 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 19:57:05.134646 ip-10-0-133-243 kubenswrapper[2578]: I0714 19:57:05.134610 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-58mq2" Jul 14 20:00:58.844080 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:00:58.844047 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:00:58.846902 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:00:58.846876 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:00:58.850639 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:00:58.850614 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:00:58.853152 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:00:58.853136 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:04:31.401601 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:31.401563 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86"] Jul 14 20:04:31.402130 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:31.401781 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" podUID="6114fb94-9b0f-4898-b470-55f87fdf371f" containerName="manager" containerID="cri-o://bbc4dd655e82d14ad81e7537b04a1126976538c53ae08804c0b86744c4913740" gracePeriod=10 Jul 14 20:04:34.448323 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.448293 2578 generic.go:356] "Generic (PLEG): container finished" podID="6114fb94-9b0f-4898-b470-55f87fdf371f" containerID="bbc4dd655e82d14ad81e7537b04a1126976538c53ae08804c0b86744c4913740" exitCode=0 Jul 14 20:04:34.448656 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.448384 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" event={"ID":"6114fb94-9b0f-4898-b470-55f87fdf371f","Type":"ContainerDied","Data":"bbc4dd655e82d14ad81e7537b04a1126976538c53ae08804c0b86744c4913740"} Jul 14 20:04:34.467763 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.467738 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 20:04:34.538328 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.538242 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mzl9\" (UniqueName: \"kubernetes.io/projected/6114fb94-9b0f-4898-b470-55f87fdf371f-kube-api-access-7mzl9\") pod \"6114fb94-9b0f-4898-b470-55f87fdf371f\" (UID: \"6114fb94-9b0f-4898-b470-55f87fdf371f\") " Jul 14 20:04:34.538493 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.538354 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6114fb94-9b0f-4898-b470-55f87fdf371f-extensions-socket-volume\") pod \"6114fb94-9b0f-4898-b470-55f87fdf371f\" (UID: \"6114fb94-9b0f-4898-b470-55f87fdf371f\") " Jul 14 20:04:34.538748 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.538726 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6114fb94-9b0f-4898-b470-55f87fdf371f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6114fb94-9b0f-4898-b470-55f87fdf371f" (UID: "6114fb94-9b0f-4898-b470-55f87fdf371f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:04:34.540340 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.540312 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6114fb94-9b0f-4898-b470-55f87fdf371f-kube-api-access-7mzl9" (OuterVolumeSpecName: "kube-api-access-7mzl9") pod "6114fb94-9b0f-4898-b470-55f87fdf371f" (UID: "6114fb94-9b0f-4898-b470-55f87fdf371f"). InnerVolumeSpecName "kube-api-access-7mzl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:04:34.639045 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.639012 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6114fb94-9b0f-4898-b470-55f87fdf371f-extensions-socket-volume\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:04:34.639045 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:34.639044 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mzl9\" (UniqueName: \"kubernetes.io/projected/6114fb94-9b0f-4898-b470-55f87fdf371f-kube-api-access-7mzl9\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:04:35.455236 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:35.455201 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" event={"ID":"6114fb94-9b0f-4898-b470-55f87fdf371f","Type":"ContainerDied","Data":"9471dff01a5b3fc88bf38243d4d185a9c8ed3119bf4805c6b1f77ffe007db6dd"} Jul 14 20:04:35.455236 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:35.455231 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86" Jul 14 20:04:35.455731 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:35.455266 2578 scope.go:117] "RemoveContainer" containerID="bbc4dd655e82d14ad81e7537b04a1126976538c53ae08804c0b86744c4913740" Jul 14 20:04:35.475019 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:35.474992 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86"] Jul 14 20:04:35.481619 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:35.481596 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5hm86"] Jul 14 20:04:36.884074 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:04:36.884037 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6114fb94-9b0f-4898-b470-55f87fdf371f" path="/var/lib/kubelet/pods/6114fb94-9b0f-4898-b470-55f87fdf371f/volumes" Jul 14 20:05:32.540501 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.540404 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l"] Jul 14 20:05:32.540987 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.540849 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6114fb94-9b0f-4898-b470-55f87fdf371f" containerName="manager" Jul 14 20:05:32.540987 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.540866 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6114fb94-9b0f-4898-b470-55f87fdf371f" containerName="manager" Jul 14 20:05:32.540987 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.540942 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6114fb94-9b0f-4898-b470-55f87fdf371f" containerName="manager" Jul 14 20:05:32.543896 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.543880 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:32.546303 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.546281 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jh6xh\"" Jul 14 20:05:32.556687 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.556663 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l"] Jul 14 20:05:32.645005 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.644962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzbr\" (UniqueName: \"kubernetes.io/projected/85fc64a4-8926-4e1b-a493-538afe904263-kube-api-access-xzzbr\") pod \"kuadrant-operator-controller-manager-86878c769f-j2z2l\" (UID: \"85fc64a4-8926-4e1b-a493-538afe904263\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:32.645191 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.645083 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85fc64a4-8926-4e1b-a493-538afe904263-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j2z2l\" (UID: \"85fc64a4-8926-4e1b-a493-538afe904263\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:32.746371 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.746335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85fc64a4-8926-4e1b-a493-538afe904263-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j2z2l\" (UID: \"85fc64a4-8926-4e1b-a493-538afe904263\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:32.746538 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.746414 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzbr\" (UniqueName: \"kubernetes.io/projected/85fc64a4-8926-4e1b-a493-538afe904263-kube-api-access-xzzbr\") pod \"kuadrant-operator-controller-manager-86878c769f-j2z2l\" (UID: \"85fc64a4-8926-4e1b-a493-538afe904263\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:32.746722 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.746699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85fc64a4-8926-4e1b-a493-538afe904263-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-j2z2l\" (UID: \"85fc64a4-8926-4e1b-a493-538afe904263\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:32.755612 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.755587 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzbr\" (UniqueName: \"kubernetes.io/projected/85fc64a4-8926-4e1b-a493-538afe904263-kube-api-access-xzzbr\") pod \"kuadrant-operator-controller-manager-86878c769f-j2z2l\" (UID: \"85fc64a4-8926-4e1b-a493-538afe904263\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:32.854235 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:32.854188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:33.003445 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:33.003406 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l"] Jul 14 20:05:33.748943 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:33.748895 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" event={"ID":"85fc64a4-8926-4e1b-a493-538afe904263","Type":"ContainerStarted","Data":"91354e811faa2113469b362fe6cbb0c0629ae2a2e9e49146400ce176e0ad49f6"} Jul 14 20:05:33.749415 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:33.748966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" event={"ID":"85fc64a4-8926-4e1b-a493-538afe904263","Type":"ContainerStarted","Data":"cc762109f705b7a85416eaf72bf4951dae73b22434bc6038017c5c629cfb88bd"} Jul 14 20:05:33.749415 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:33.749106 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:33.777285 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:33.777233 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" podStartSLOduration=1.777216337 podStartE2EDuration="1.777216337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:05:33.774622859 +0000 UTC m=+1175.422186425" watchObservedRunningTime="2026-07-14 20:05:33.777216337 +0000 UTC m=+1175.424779902" Jul 14 20:05:44.756578 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:44.756484 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-j2z2l" Jul 14 20:05:58.875623 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:58.875509 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:05:58.888513 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:58.879277 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:05:58.888513 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:58.882961 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:05:58.888513 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:05:58.886405 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:10:58.907077 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:10:58.906965 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:10:58.911170 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:10:58.911146 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:10:58.912975 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:10:58.912955 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:10:58.916980 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:10:58.916962 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:12:13.563895 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.563856 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg"] Jul 14 20:12:13.567505 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.567484 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" Jul 14 20:12:13.570033 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.570008 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:12:13.570139 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.570040 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:12:13.573945 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.573923 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg"] Jul 14 20:12:13.646554 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.646521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448882c-f190-40f6-bc80-9010e2d9a768-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg\" (UID: \"7448882c-f190-40f6-bc80-9010e2d9a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" Jul 14 20:12:13.747890 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.747858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448882c-f190-40f6-bc80-9010e2d9a768-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg\" (UID: \"7448882c-f190-40f6-bc80-9010e2d9a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" Jul 14 20:12:13.748533 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.748511 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448882c-f190-40f6-bc80-9010e2d9a768-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg\" (UID: \"7448882c-f190-40f6-bc80-9010e2d9a768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" Jul 14 20:12:13.878970 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:13.878881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" Jul 14 20:12:14.221163 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:14.221133 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg"] Jul 14 20:12:14.223662 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:12:14.223632 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7448882c_f190_40f6_bc80_9010e2d9a768.slice/crio-a791aaf0fa6d4f0f68378e99067bf6b35961063f822f2e4655e034bb9bbddc07 WatchSource:0}: Error finding container a791aaf0fa6d4f0f68378e99067bf6b35961063f822f2e4655e034bb9bbddc07: Status 404 returned error can't find the container with id a791aaf0fa6d4f0f68378e99067bf6b35961063f822f2e4655e034bb9bbddc07 Jul 14 20:12:14.225715 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:14.225697 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:12:14.849680 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:14.849643 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" event={"ID":"7448882c-f190-40f6-bc80-9010e2d9a768","Type":"ContainerStarted","Data":"a791aaf0fa6d4f0f68378e99067bf6b35961063f822f2e4655e034bb9bbddc07"} Jul 14 20:12:16.861001 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:16.860964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" event={"ID":"7448882c-f190-40f6-bc80-9010e2d9a768","Type":"ContainerStarted","Data":"6949e581058d4e0cbf57f7aee5f81fce08227a5cc85803042527fc51950f6bf7"} Jul 14 20:12:16.882683 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:16.878668 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" podStartSLOduration=1.769313533 podStartE2EDuration="3.878645778s" totalImagesPullingTime="2.109332245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:12:16.876735913 +0000 UTC m=+1578.524299478" watchObservedRunningTime="2026-07-14 20:12:16.878645778 +0000 UTC m=+1578.526209344" Jul 14 20:12:17.867254 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:17.867215 2578 generic.go:356] "Generic (PLEG): container finished" podID="7448882c-f190-40f6-bc80-9010e2d9a768" containerID="6949e581058d4e0cbf57f7aee5f81fce08227a5cc85803042527fc51950f6bf7" exitCode=7 Jul 14 20:12:17.867631 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:17.867287 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" event={"ID":"7448882c-f190-40f6-bc80-9010e2d9a768","Type":"ContainerDied","Data":"6949e581058d4e0cbf57f7aee5f81fce08227a5cc85803042527fc51950f6bf7"} Jul 14 20:12:17.867631 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:17.867446 2578 scope.go:117] "RemoveContainer" containerID="6949e581058d4e0cbf57f7aee5f81fce08227a5cc85803042527fc51950f6bf7" Jul 14 20:12:18.873795 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:18.873759 2578 generic.go:356] "Generic (PLEG): container finished" podID="7448882c-f190-40f6-bc80-9010e2d9a768" containerID="341673bbab455f7192a62aefe1a90ab7f8e97ea60db39b07ff2b6a0ef40443af" exitCode=0 Jul 14 20:12:18.874213 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:18.873858 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" event={"ID":"7448882c-f190-40f6-bc80-9010e2d9a768","Type":"ContainerDied","Data":"341673bbab455f7192a62aefe1a90ab7f8e97ea60db39b07ff2b6a0ef40443af"} Jul 14 20:12:18.874213 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:18.873928 2578 scope.go:117] "RemoveContainer" containerID="6949e581058d4e0cbf57f7aee5f81fce08227a5cc85803042527fc51950f6bf7" Jul 14 20:12:20.023652 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:20.023626 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" Jul 14 20:12:20.104728 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:20.104690 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448882c-f190-40f6-bc80-9010e2d9a768-maas-api-service-ca\") pod \"7448882c-f190-40f6-bc80-9010e2d9a768\" (UID: \"7448882c-f190-40f6-bc80-9010e2d9a768\") " Jul 14 20:12:20.105122 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:20.105095 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7448882c-f190-40f6-bc80-9010e2d9a768-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7448882c-f190-40f6-bc80-9010e2d9a768" (UID: "7448882c-f190-40f6-bc80-9010e2d9a768"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:12:20.205481 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:20.205396 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7448882c-f190-40f6-bc80-9010e2d9a768-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:12:20.886249 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:20.886216 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" Jul 14 20:12:20.886450 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:20.886256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg" event={"ID":"7448882c-f190-40f6-bc80-9010e2d9a768","Type":"ContainerDied","Data":"a791aaf0fa6d4f0f68378e99067bf6b35961063f822f2e4655e034bb9bbddc07"} Jul 14 20:12:20.886450 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:20.886302 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a791aaf0fa6d4f0f68378e99067bf6b35961063f822f2e4655e034bb9bbddc07" Jul 14 20:12:39.395517 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.395483 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg"] Jul 14 20:12:39.396009 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.395950 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7448882c-f190-40f6-bc80-9010e2d9a768" containerName="revoke-keys" Jul 14 20:12:39.396009 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.395964 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7448882c-f190-40f6-bc80-9010e2d9a768" containerName="revoke-keys" Jul 14 20:12:39.396009 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.395973 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7448882c-f190-40f6-bc80-9010e2d9a768" containerName="revoke-keys" Jul 14 20:12:39.396009 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.395978 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7448882c-f190-40f6-bc80-9010e2d9a768" containerName="revoke-keys" Jul 14 20:12:39.396142 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.396062 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7448882c-f190-40f6-bc80-9010e2d9a768" containerName="revoke-keys" Jul 14 20:12:39.396142 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.396070 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7448882c-f190-40f6-bc80-9010e2d9a768" containerName="revoke-keys" Jul 14 20:12:39.399174 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.399151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.402260 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.402149 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f4a7e811-gw-openshift-default-dockercfg-mjm4x\"" Jul 14 20:12:39.414126 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.414102 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg"] Jul 14 20:12:39.487157 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-envoy\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.487335 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-podinfo\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.487335 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-socket\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.487335 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487281 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-credential-socket\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.487335 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487297 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-certs\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.487335 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-token\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.487533 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-data\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.487533 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brxlx\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-kube-api-access-brxlx\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.487533 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.487446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istiod-ca-cert\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588123 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-socket\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588312 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-credential-socket\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588312 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-certs\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588394 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-token\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588394 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-data\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588498 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brxlx\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-kube-api-access-brxlx\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588498 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istiod-ca-cert\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588498 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-envoy\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588645 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-podinfo\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588645 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588583 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-credential-socket\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588645 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-socket\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588775 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588738 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-data\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.588822 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.588772 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-certs\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.589116 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.589097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istiod-ca-cert\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.590882 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.590859 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-envoy\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.590997 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.590960 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-podinfo\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.598060 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.598037 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-token\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.598496 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.598475 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brxlx\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-kube-api-access-brxlx\") pod \"e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.711977 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.711882 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:39.862809 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.862784 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg"] Jul 14 20:12:39.865086 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:12:39.865058 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f40d28_4ce7_4426_8909_19e6c3f3a85f.slice/crio-b7113d383866f22cfb5827277ff21a74850928cc5ed1a8a2de593b3dd1d44144 WatchSource:0}: Error finding container b7113d383866f22cfb5827277ff21a74850928cc5ed1a8a2de593b3dd1d44144: Status 404 returned error can't find the container with id b7113d383866f22cfb5827277ff21a74850928cc5ed1a8a2de593b3dd1d44144 Jul 14 20:12:39.867287 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.867253 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:39.867393 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.867321 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:39.867393 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.867352 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:39.986170 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.986078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" event={"ID":"a8f40d28-4ce7-4426-8909-19e6c3f3a85f","Type":"ContainerStarted","Data":"24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b"} Jul 14 20:12:39.986170 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:39.986136 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" event={"ID":"a8f40d28-4ce7-4426-8909-19e6c3f3a85f","Type":"ContainerStarted","Data":"b7113d383866f22cfb5827277ff21a74850928cc5ed1a8a2de593b3dd1d44144"} Jul 14 20:12:40.008502 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:40.008444 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" podStartSLOduration=1.008422245 podStartE2EDuration="1.008422245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:12:40.0069609 +0000 UTC m=+1601.654524479" watchObservedRunningTime="2026-07-14 20:12:40.008422245 +0000 UTC m=+1601.655985794" Jul 14 20:12:40.713105 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:40.713065 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:40.718042 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:40.718016 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:40.991854 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:40.991752 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:40.992628 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:40.992604 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:56.553089 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:56.552517 2578 patch_prober.go:28] interesting pod/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:56874->10.132.0.48:15021: read: connection reset by peer" start-of-body= Jul 14 20:12:56.553089 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:56.552581 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:56874->10.132.0.48:15021: read: connection reset by peer" Jul 14 20:12:57.092847 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:57.092805 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerID="24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b" exitCode=0 Jul 14 20:12:57.093034 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:57.092878 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" event={"ID":"a8f40d28-4ce7-4426-8909-19e6c3f3a85f","Type":"ContainerDied","Data":"24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b"} Jul 14 20:12:57.093306 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:57.093283 2578 scope.go:117] "RemoveContainer" containerID="24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b" Jul 14 20:12:57.094322 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:57.094295 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:57.094377 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:57.094364 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:57.094414 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:57.094399 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 14 20:12:58.100352 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:58.100318 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" event={"ID":"a8f40d28-4ce7-4426-8909-19e6c3f3a85f","Type":"ContainerStarted","Data":"cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768"} Jul 14 20:12:58.712306 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:58.712269 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:12:58.712515 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:58.712493 2578 patch_prober.go:28] interesting pod/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:12:58.712569 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:58.712543 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 14 20:12:59.712378 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:59.712336 2578 patch_prober.go:28] interesting pod/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:12:59.712799 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:12:59.712405 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 14 20:13:00.712737 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:00.712695 2578 patch_prober.go:28] interesting pod/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:13:00.713163 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:00.712757 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 14 20:13:01.713359 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:01.713308 2578 patch_prober.go:28] interesting pod/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 14 20:13:01.713769 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:01.713377 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 14 20:13:02.716492 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:02.716460 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:13:02.716958 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:02.716733 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:13:02.717402 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:02.717384 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:13:45.293666 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:45.293626 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg"] Jul 14 20:13:45.294291 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:45.293844 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" containerID="cri-o://cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768" gracePeriod=30 Jul 14 20:13:47.719602 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:47.719562 2578 patch_prober.go:28] interesting pod/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 20:13:47.720105 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:47.719633 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 20:13:50.545120 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.545094 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:13:50.639539 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639513 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istiod-ca-cert\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.639691 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639579 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-data\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.639691 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-envoy\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.639691 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639627 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-credential-socket\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.639691 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639658 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-podinfo\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.639943 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639702 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-socket\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.639943 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639743 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-token\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.639943 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639767 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brxlx\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-kube-api-access-brxlx\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.639943 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639860 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-certs\") pod \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\" (UID: \"a8f40d28-4ce7-4426-8909-19e6c3f3a85f\") " Jul 14 20:13:50.640153 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.639971 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:13:50.640235 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.640213 2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istiod-ca-cert\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:50.640586 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.640561 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:13:50.640735 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.640711 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:13:50.640809 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.640774 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:13:50.641634 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.641604 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:13:50.642431 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.642404 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-kube-api-access-brxlx" (OuterVolumeSpecName: "kube-api-access-brxlx") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "kube-api-access-brxlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:13:50.642528 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.642443 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:13:50.642776 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.642755 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 20:13:50.643439 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.643414 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8f40d28-4ce7-4426-8909-19e6c3f3a85f" (UID: "a8f40d28-4ce7-4426-8909-19e6c3f3a85f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:13:50.740890 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.740842 2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-envoy\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:50.740890 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.740887 2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-credential-socket\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:50.740890 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.740898 2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-podinfo\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:50.740890 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.740907 2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-socket\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:50.741288 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.740916 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-token\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:50.741288 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.740928 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brxlx\" (UniqueName: \"kubernetes.io/projected/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-kube-api-access-brxlx\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:50.741288 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.740938 2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-workload-certs\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:50.741288 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:50.740947 2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8f40d28-4ce7-4426-8909-19e6c3f3a85f-istio-data\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:13:51.387155 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.387120 2578 generic.go:356] "Generic (PLEG): container finished" podID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerID="cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768" exitCode=0 Jul 14 20:13:51.387155 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.387161 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" event={"ID":"a8f40d28-4ce7-4426-8909-19e6c3f3a85f","Type":"ContainerDied","Data":"cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768"} Jul 14 20:13:51.387418 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.387181 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" Jul 14 20:13:51.387418 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.387206 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg" event={"ID":"a8f40d28-4ce7-4426-8909-19e6c3f3a85f","Type":"ContainerDied","Data":"b7113d383866f22cfb5827277ff21a74850928cc5ed1a8a2de593b3dd1d44144"} Jul 14 20:13:51.387418 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.387228 2578 scope.go:117] "RemoveContainer" containerID="cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768" Jul 14 20:13:51.396428 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.396394 2578 scope.go:117] "RemoveContainer" containerID="24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b" Jul 14 20:13:51.405024 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.405005 2578 scope.go:117] "RemoveContainer" containerID="cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768" Jul 14 20:13:51.405312 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:13:51.405289 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768\": container with ID starting with cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768 not found: ID does not exist" containerID="cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768" Jul 14 20:13:51.405364 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.405326 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768"} err="failed to get container status \"cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768\": rpc error: code = NotFound desc = could not find container \"cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768\": container with ID starting with cf21a89a5f621db0b7d6a550e90621ec3e431ed21ac0c1bb44b4e2298ef8b768 not found: ID does not exist" Jul 14 20:13:51.405364 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.405348 2578 scope.go:117] "RemoveContainer" containerID="24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b" Jul 14 20:13:51.405582 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:13:51.405560 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b\": container with ID starting with 24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b not found: ID does not exist" containerID="24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b" Jul 14 20:13:51.405660 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.405593 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b"} err="failed to get container status \"24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b\": rpc error: code = NotFound desc = could not find container \"24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b\": container with ID starting with 24c71eeb87558b018871b5ec066b6ffb101ddba63aa1e6976c8a07cf169e453b not found: ID does not exist" Jul 14 20:13:51.414435 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.414390 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg"] Jul 14 20:13:51.433850 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:51.424532 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f4a7e811-gw-openshift-default-65ff467f87-52rbg"] Jul 14 20:13:52.886193 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:13:52.886145 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" path="/var/lib/kubelet/pods/a8f40d28-4ce7-4426-8909-19e6c3f3a85f/volumes" Jul 14 20:14:04.404687 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.404642 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c"] Jul 14 20:14:04.405314 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.405291 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" Jul 14 20:14:04.405404 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.405317 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" Jul 14 20:14:04.405404 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.405342 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" Jul 14 20:14:04.405404 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.405352 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" Jul 14 20:14:04.405551 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.405462 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" Jul 14 20:14:04.408653 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.408629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" Jul 14 20:14:04.411605 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.411581 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:14:04.411814 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.411648 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:14:04.416885 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.416750 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c"] Jul 14 20:14:04.473972 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.473936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1820935a-b3b8-423a-a863-92a8d2ae5c5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c\" (UID: \"1820935a-b3b8-423a-a863-92a8d2ae5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" Jul 14 20:14:04.574930 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.574885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1820935a-b3b8-423a-a863-92a8d2ae5c5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c\" (UID: \"1820935a-b3b8-423a-a863-92a8d2ae5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" Jul 14 20:14:04.575553 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.575528 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1820935a-b3b8-423a-a863-92a8d2ae5c5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c\" (UID: \"1820935a-b3b8-423a-a863-92a8d2ae5c5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" Jul 14 20:14:04.721404 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.721310 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" Jul 14 20:14:04.862071 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:04.862047 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c"] Jul 14 20:14:04.864285 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:14:04.864260 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1820935a_b3b8_423a_a863_92a8d2ae5c5c.slice/crio-a745a60e3eb42173c08d673d847cec797bd209102f3ba422a023d4db16c96d39 WatchSource:0}: Error finding container a745a60e3eb42173c08d673d847cec797bd209102f3ba422a023d4db16c96d39: Status 404 returned error can't find the container with id a745a60e3eb42173c08d673d847cec797bd209102f3ba422a023d4db16c96d39 Jul 14 20:14:05.463097 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:05.463065 2578 generic.go:356] "Generic (PLEG): container finished" podID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerID="185222460e507825c7afc95c3a2ee513ae763c3e5773fa7be8508d740e3959de" exitCode=7 Jul 14 20:14:05.463518 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:05.463144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" event={"ID":"1820935a-b3b8-423a-a863-92a8d2ae5c5c","Type":"ContainerDied","Data":"185222460e507825c7afc95c3a2ee513ae763c3e5773fa7be8508d740e3959de"} Jul 14 20:14:05.463518 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:05.463198 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" event={"ID":"1820935a-b3b8-423a-a863-92a8d2ae5c5c","Type":"ContainerStarted","Data":"a745a60e3eb42173c08d673d847cec797bd209102f3ba422a023d4db16c96d39"} Jul 14 20:14:05.463518 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:05.463337 2578 scope.go:117] "RemoveContainer" containerID="185222460e507825c7afc95c3a2ee513ae763c3e5773fa7be8508d740e3959de" Jul 14 20:14:06.471209 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:06.471116 2578 generic.go:356] "Generic (PLEG): container finished" podID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerID="36b7a17620733201b6ad79ae9b175646c0ef54a4af894ec0dd51d9ef726bd61e" exitCode=7 Jul 14 20:14:06.471598 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:06.471207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" event={"ID":"1820935a-b3b8-423a-a863-92a8d2ae5c5c","Type":"ContainerDied","Data":"36b7a17620733201b6ad79ae9b175646c0ef54a4af894ec0dd51d9ef726bd61e"} Jul 14 20:14:06.471598 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:06.471266 2578 scope.go:117] "RemoveContainer" containerID="185222460e507825c7afc95c3a2ee513ae763c3e5773fa7be8508d740e3959de" Jul 14 20:14:06.471598 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:06.471451 2578 scope.go:117] "RemoveContainer" containerID="36b7a17620733201b6ad79ae9b175646c0ef54a4af894ec0dd51d9ef726bd61e" Jul 14 20:14:06.471736 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:14:06.471687 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c_odh-ai-gateway-infra(1820935a-b3b8-423a-a863-92a8d2ae5c5c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" Jul 14 20:14:07.478988 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:07.478952 2578 scope.go:117] "RemoveContainer" containerID="36b7a17620733201b6ad79ae9b175646c0ef54a4af894ec0dd51d9ef726bd61e" Jul 14 20:14:07.479387 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:14:07.479196 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c_odh-ai-gateway-infra(1820935a-b3b8-423a-a863-92a8d2ae5c5c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" Jul 14 20:14:22.878486 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:22.878396 2578 scope.go:117] "RemoveContainer" containerID="36b7a17620733201b6ad79ae9b175646c0ef54a4af894ec0dd51d9ef726bd61e" Jul 14 20:14:23.572297 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:23.572265 2578 generic.go:356] "Generic (PLEG): container finished" podID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerID="77fbcb870155bc52c2a9831b4adabbe5d3ec57d21532746a56ebeb6d045175d6" exitCode=0 Jul 14 20:14:23.572465 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:23.572336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" event={"ID":"1820935a-b3b8-423a-a863-92a8d2ae5c5c","Type":"ContainerDied","Data":"77fbcb870155bc52c2a9831b4adabbe5d3ec57d21532746a56ebeb6d045175d6"} Jul 14 20:14:23.572465 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:23.572405 2578 scope.go:117] "RemoveContainer" containerID="36b7a17620733201b6ad79ae9b175646c0ef54a4af894ec0dd51d9ef726bd61e" Jul 14 20:14:24.606611 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:24.606575 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c"] Jul 14 20:14:24.724508 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:24.724478 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" Jul 14 20:14:24.767054 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:24.767021 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1820935a-b3b8-423a-a863-92a8d2ae5c5c-maas-api-service-ca\") pod \"1820935a-b3b8-423a-a863-92a8d2ae5c5c\" (UID: \"1820935a-b3b8-423a-a863-92a8d2ae5c5c\") " Jul 14 20:14:24.767357 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:24.767336 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1820935a-b3b8-423a-a863-92a8d2ae5c5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1820935a-b3b8-423a-a863-92a8d2ae5c5c" (UID: "1820935a-b3b8-423a-a863-92a8d2ae5c5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:14:24.868235 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:24.868138 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1820935a-b3b8-423a-a863-92a8d2ae5c5c-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:14:25.587181 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:25.587152 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" Jul 14 20:14:25.587372 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:25.587205 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c" event={"ID":"1820935a-b3b8-423a-a863-92a8d2ae5c5c","Type":"ContainerDied","Data":"a745a60e3eb42173c08d673d847cec797bd209102f3ba422a023d4db16c96d39"} Jul 14 20:14:25.587372 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:25.587266 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a745a60e3eb42173c08d673d847cec797bd209102f3ba422a023d4db16c96d39" Jul 14 20:14:25.617152 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:25.617120 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c"] Jul 14 20:14:25.621484 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:25.621460 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-8sz2c"] Jul 14 20:14:26.884168 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:14:26.884134 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" path="/var/lib/kubelet/pods/1820935a-b3b8-423a-a863-92a8d2ae5c5c/volumes" Jul 14 20:15:00.148374 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.148329 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp"] Jul 14 20:15:00.149048 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.148986 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.149048 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.149007 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.149048 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.149036 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.149048 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.149044 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.149274 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.149154 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f40d28-4ce7-4426-8909-19e6c3f3a85f" containerName="istio-proxy" Jul 14 20:15:00.149274 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.149167 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.149274 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.149179 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.149274 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.149188 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.152527 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.152503 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c"] Jul 14 20:15:00.152654 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.152624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" Jul 14 20:15:00.153073 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.153048 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.153169 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.153075 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1820935a-b3b8-423a-a863-92a8d2ae5c5c" containerName="revoke-keys" Jul 14 20:15:00.155022 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.155006 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:15:00.156487 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.156466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c" Jul 14 20:15:00.160576 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.160546 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp"] Jul 14 20:15:00.164259 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.164228 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c"] Jul 14 20:15:00.167132 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.167108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" Jul 14 20:15:00.175166 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.175125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c" Jul 14 20:15:00.331418 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.331392 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp"] Jul 14 20:15:00.334520 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:15:00.334490 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddefaeae_898d_430d_aec8_2ff4df478ab1.slice/crio-80bdfeb815208a75899d5f6f31e3f112a5e724887981858674c2a6b7d8723e8c WatchSource:0}: Error finding container 80bdfeb815208a75899d5f6f31e3f112a5e724887981858674c2a6b7d8723e8c: Status 404 returned error can't find the container with id 80bdfeb815208a75899d5f6f31e3f112a5e724887981858674c2a6b7d8723e8c Jul 14 20:15:00.563313 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.563297 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c"] Jul 14 20:15:00.565482 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:15:00.565458 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a90cce_d2bb_4f2f_a3aa_c59b8f44d9b4.slice/crio-2bc9c796e3a4ea5e86010e7d6b605ef0315ba6532090d480d3d4118c9d88550b WatchSource:0}: Error finding container 2bc9c796e3a4ea5e86010e7d6b605ef0315ba6532090d480d3d4118c9d88550b: Status 404 returned error can't find the container with id 2bc9c796e3a4ea5e86010e7d6b605ef0315ba6532090d480d3d4118c9d88550b Jul 14 20:15:00.767779 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.767742 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" event={"ID":"ddefaeae-898d-430d-aec8-2ff4df478ab1","Type":"ContainerStarted","Data":"a83aaf89f08898d6d4f469b8cb1e069c0313d4a919bc20e388726f891b1c7f86"} Jul 14 20:15:00.767986 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.767809 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" event={"ID":"ddefaeae-898d-430d-aec8-2ff4df478ab1","Type":"ContainerStarted","Data":"80bdfeb815208a75899d5f6f31e3f112a5e724887981858674c2a6b7d8723e8c"} Jul 14 20:15:00.769029 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.769004 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c" event={"ID":"a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4","Type":"ContainerStarted","Data":"2bc9c796e3a4ea5e86010e7d6b605ef0315ba6532090d480d3d4118c9d88550b"} Jul 14 20:15:00.785951 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:00.785900 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" podStartSLOduration=0.78588464 podStartE2EDuration="785.88464ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:15:00.785002735 +0000 UTC m=+1742.432566314" watchObservedRunningTime="2026-07-14 20:15:00.78588464 +0000 UTC m=+1742.433448205" Jul 14 20:15:01.776222 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:01.776184 2578 generic.go:356] "Generic (PLEG): container finished" podID="a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4" containerID="3b73619b995bf3f3ed85d57ee06713470b875f37124339f379e891abd98981f5" exitCode=0 Jul 14 20:15:01.776691 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:01.776267 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c" event={"ID":"a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4","Type":"ContainerDied","Data":"3b73619b995bf3f3ed85d57ee06713470b875f37124339f379e891abd98981f5"} Jul 14 20:15:01.777771 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:01.777746 2578 generic.go:356] "Generic (PLEG): container finished" podID="ddefaeae-898d-430d-aec8-2ff4df478ab1" containerID="a83aaf89f08898d6d4f469b8cb1e069c0313d4a919bc20e388726f891b1c7f86" exitCode=0 Jul 14 20:15:01.777914 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:01.777792 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" event={"ID":"ddefaeae-898d-430d-aec8-2ff4df478ab1","Type":"ContainerDied","Data":"a83aaf89f08898d6d4f469b8cb1e069c0313d4a919bc20e388726f891b1c7f86"} Jul 14 20:15:02.933759 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:02.933740 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c" Jul 14 20:15:02.956795 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:02.956767 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" Jul 14 20:15:03.796106 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:03.796069 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" Jul 14 20:15:03.796106 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:03.796088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734335-879vp" event={"ID":"ddefaeae-898d-430d-aec8-2ff4df478ab1","Type":"ContainerDied","Data":"80bdfeb815208a75899d5f6f31e3f112a5e724887981858674c2a6b7d8723e8c"} Jul 14 20:15:03.796352 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:03.796175 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80bdfeb815208a75899d5f6f31e3f112a5e724887981858674c2a6b7d8723e8c" Jul 14 20:15:03.797624 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:03.797589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c" event={"ID":"a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4","Type":"ContainerDied","Data":"2bc9c796e3a4ea5e86010e7d6b605ef0315ba6532090d480d3d4118c9d88550b"} Jul 14 20:15:03.797743 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:03.797643 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bc9c796e3a4ea5e86010e7d6b605ef0315ba6532090d480d3d4118c9d88550b" Jul 14 20:15:03.797743 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:03.797604 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c" Jul 14 20:15:04.485614 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.485576 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7"] Jul 14 20:15:04.486045 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.486032 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4" containerName="cleanup" Jul 14 20:15:04.486091 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.486047 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4" containerName="cleanup" Jul 14 20:15:04.486091 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.486065 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddefaeae-898d-430d-aec8-2ff4df478ab1" containerName="cleanup" Jul 14 20:15:04.486091 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.486071 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddefaeae-898d-430d-aec8-2ff4df478ab1" containerName="cleanup" Jul 14 20:15:04.486189 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.486143 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4" containerName="cleanup" Jul 14 20:15:04.486189 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.486153 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddefaeae-898d-430d-aec8-2ff4df478ab1" containerName="cleanup" Jul 14 20:15:04.489107 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.489092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" Jul 14 20:15:04.491887 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.491854 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:15:04.492043 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.491919 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:15:04.500688 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.500662 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7"] Jul 14 20:15:04.636894 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.636857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eb0459-918c-4a9d-87b9-16cc92d00d6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7\" (UID: \"32eb0459-918c-4a9d-87b9-16cc92d00d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" Jul 14 20:15:04.738493 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.738399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eb0459-918c-4a9d-87b9-16cc92d00d6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7\" (UID: \"32eb0459-918c-4a9d-87b9-16cc92d00d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" Jul 14 20:15:04.739116 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.739098 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eb0459-918c-4a9d-87b9-16cc92d00d6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7\" (UID: \"32eb0459-918c-4a9d-87b9-16cc92d00d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" Jul 14 20:15:04.800663 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.800634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" Jul 14 20:15:04.947305 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:04.947270 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7"] Jul 14 20:15:04.948940 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:15:04.948899 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32eb0459_918c_4a9d_87b9_16cc92d00d6f.slice/crio-d8abbc96417d6fecca1e1701b7a8a1e5aef5d554f6b79838b54be29f904d280b WatchSource:0}: Error finding container d8abbc96417d6fecca1e1701b7a8a1e5aef5d554f6b79838b54be29f904d280b: Status 404 returned error can't find the container with id d8abbc96417d6fecca1e1701b7a8a1e5aef5d554f6b79838b54be29f904d280b Jul 14 20:15:05.811162 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:05.811122 2578 generic.go:356] "Generic (PLEG): container finished" podID="32eb0459-918c-4a9d-87b9-16cc92d00d6f" containerID="fdb857704dfa7737ae609c61e70f36290b712e3bc3544b08e0514abc43d49f1d" exitCode=0 Jul 14 20:15:05.811534 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:05.811207 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" event={"ID":"32eb0459-918c-4a9d-87b9-16cc92d00d6f","Type":"ContainerDied","Data":"fdb857704dfa7737ae609c61e70f36290b712e3bc3544b08e0514abc43d49f1d"} Jul 14 20:15:05.811534 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:05.811264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" event={"ID":"32eb0459-918c-4a9d-87b9-16cc92d00d6f","Type":"ContainerStarted","Data":"d8abbc96417d6fecca1e1701b7a8a1e5aef5d554f6b79838b54be29f904d280b"} Jul 14 20:15:06.959068 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:06.959041 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" Jul 14 20:15:06.959467 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:06.959450 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eb0459-918c-4a9d-87b9-16cc92d00d6f-maas-api-service-ca\") pod \"32eb0459-918c-4a9d-87b9-16cc92d00d6f\" (UID: \"32eb0459-918c-4a9d-87b9-16cc92d00d6f\") " Jul 14 20:15:06.959807 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:06.959786 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32eb0459-918c-4a9d-87b9-16cc92d00d6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32eb0459-918c-4a9d-87b9-16cc92d00d6f" (UID: "32eb0459-918c-4a9d-87b9-16cc92d00d6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:15:06.959912 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:06.959897 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32eb0459-918c-4a9d-87b9-16cc92d00d6f-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:15:07.822507 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:07.822476 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" Jul 14 20:15:07.822687 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:07.822476 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7" event={"ID":"32eb0459-918c-4a9d-87b9-16cc92d00d6f","Type":"ContainerDied","Data":"d8abbc96417d6fecca1e1701b7a8a1e5aef5d554f6b79838b54be29f904d280b"} Jul 14 20:15:07.822687 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:07.822596 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8abbc96417d6fecca1e1701b7a8a1e5aef5d554f6b79838b54be29f904d280b" Jul 14 20:15:15.085062 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:15.085019 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c"] Jul 14 20:15:15.096789 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:15.096755 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f61d89f4-29734335-78c8c"] Jul 14 20:15:16.883439 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:16.883401 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4" path="/var/lib/kubelet/pods/a0a90cce-d2bb-4f2f-a3aa-c59b8f44d9b4/volumes" Jul 14 20:15:36.475021 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.474979 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx"] Jul 14 20:15:36.475458 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.475408 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32eb0459-918c-4a9d-87b9-16cc92d00d6f" containerName="revoke-keys" Jul 14 20:15:36.475458 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.475421 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32eb0459-918c-4a9d-87b9-16cc92d00d6f" containerName="revoke-keys" Jul 14 20:15:36.475533 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.475481 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="32eb0459-918c-4a9d-87b9-16cc92d00d6f" containerName="revoke-keys" Jul 14 20:15:36.479230 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.479212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" Jul 14 20:15:36.482779 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.482747 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:15:36.482779 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.482762 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:15:36.484533 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.484507 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx"] Jul 14 20:15:36.619905 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.619861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776c3eb-91f7-46bb-b499-e4a5322bd042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx\" (UID: \"d776c3eb-91f7-46bb-b499-e4a5322bd042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" Jul 14 20:15:36.721175 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.721144 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776c3eb-91f7-46bb-b499-e4a5322bd042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx\" (UID: \"d776c3eb-91f7-46bb-b499-e4a5322bd042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" Jul 14 20:15:36.721818 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.721788 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776c3eb-91f7-46bb-b499-e4a5322bd042-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx\" (UID: \"d776c3eb-91f7-46bb-b499-e4a5322bd042\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" Jul 14 20:15:36.791349 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.791249 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" Jul 14 20:15:36.940074 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.939874 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx"] Jul 14 20:15:36.942712 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:15:36.942673 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd776c3eb_91f7_46bb_b499_e4a5322bd042.slice/crio-f3d4313c1c485e0634fa17e8f9f253737823a876e4dd713ad715f11459072f48 WatchSource:0}: Error finding container f3d4313c1c485e0634fa17e8f9f253737823a876e4dd713ad715f11459072f48: Status 404 returned error can't find the container with id f3d4313c1c485e0634fa17e8f9f253737823a876e4dd713ad715f11459072f48 Jul 14 20:15:36.980199 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:36.980169 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" event={"ID":"d776c3eb-91f7-46bb-b499-e4a5322bd042","Type":"ContainerStarted","Data":"f3d4313c1c485e0634fa17e8f9f253737823a876e4dd713ad715f11459072f48"} Jul 14 20:15:37.986740 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:37.986641 2578 generic.go:356] "Generic (PLEG): container finished" podID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerID="f73fed6ba10d1998238be1da7e742d3e1d4b23efecc99b95cb917dfa1acd445a" exitCode=7 Jul 14 20:15:37.987140 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:37.986729 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" event={"ID":"d776c3eb-91f7-46bb-b499-e4a5322bd042","Type":"ContainerDied","Data":"f73fed6ba10d1998238be1da7e742d3e1d4b23efecc99b95cb917dfa1acd445a"} Jul 14 20:15:37.987140 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:37.986930 2578 scope.go:117] "RemoveContainer" containerID="f73fed6ba10d1998238be1da7e742d3e1d4b23efecc99b95cb917dfa1acd445a" Jul 14 20:15:38.994465 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:38.994428 2578 generic.go:356] "Generic (PLEG): container finished" podID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerID="22916912ab9e573b0d4350f1a487d17bf79b63ed4ad3eb3fe7bf611d293f9147" exitCode=7 Jul 14 20:15:38.994934 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:38.994503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" event={"ID":"d776c3eb-91f7-46bb-b499-e4a5322bd042","Type":"ContainerDied","Data":"22916912ab9e573b0d4350f1a487d17bf79b63ed4ad3eb3fe7bf611d293f9147"} Jul 14 20:15:38.994934 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:38.994559 2578 scope.go:117] "RemoveContainer" containerID="f73fed6ba10d1998238be1da7e742d3e1d4b23efecc99b95cb917dfa1acd445a" Jul 14 20:15:38.994934 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:38.994759 2578 scope.go:117] "RemoveContainer" containerID="22916912ab9e573b0d4350f1a487d17bf79b63ed4ad3eb3fe7bf611d293f9147" Jul 14 20:15:38.995140 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:15:38.995058 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx_odh-ai-gateway-infra(d776c3eb-91f7-46bb-b499-e4a5322bd042)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" Jul 14 20:15:40.001563 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:40.001535 2578 scope.go:117] "RemoveContainer" containerID="22916912ab9e573b0d4350f1a487d17bf79b63ed4ad3eb3fe7bf611d293f9147" Jul 14 20:15:40.002061 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:15:40.001728 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx_odh-ai-gateway-infra(d776c3eb-91f7-46bb-b499-e4a5322bd042)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" Jul 14 20:15:50.878225 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:50.878193 2578 scope.go:117] "RemoveContainer" containerID="22916912ab9e573b0d4350f1a487d17bf79b63ed4ad3eb3fe7bf611d293f9147" Jul 14 20:15:52.065068 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:52.065034 2578 generic.go:356] "Generic (PLEG): container finished" podID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerID="d915422b5e615e03c1509b8fb7b5683c4b58d16285eff1c311096b9aebe4b2b8" exitCode=0 Jul 14 20:15:52.065456 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:52.065103 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" event={"ID":"d776c3eb-91f7-46bb-b499-e4a5322bd042","Type":"ContainerDied","Data":"d915422b5e615e03c1509b8fb7b5683c4b58d16285eff1c311096b9aebe4b2b8"} Jul 14 20:15:52.065456 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:52.065166 2578 scope.go:117] "RemoveContainer" containerID="22916912ab9e573b0d4350f1a487d17bf79b63ed4ad3eb3fe7bf611d293f9147" Jul 14 20:15:53.108796 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:53.108754 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx"] Jul 14 20:15:53.215981 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:53.215957 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" Jul 14 20:15:53.376270 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:53.376170 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776c3eb-91f7-46bb-b499-e4a5322bd042-maas-api-service-ca\") pod \"d776c3eb-91f7-46bb-b499-e4a5322bd042\" (UID: \"d776c3eb-91f7-46bb-b499-e4a5322bd042\") " Jul 14 20:15:53.376640 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:53.376606 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d776c3eb-91f7-46bb-b499-e4a5322bd042-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d776c3eb-91f7-46bb-b499-e4a5322bd042" (UID: "d776c3eb-91f7-46bb-b499-e4a5322bd042"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:15:53.477285 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:53.477230 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d776c3eb-91f7-46bb-b499-e4a5322bd042-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:15:54.079352 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:54.079319 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" Jul 14 20:15:54.079527 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:54.079352 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx" event={"ID":"d776c3eb-91f7-46bb-b499-e4a5322bd042","Type":"ContainerDied","Data":"f3d4313c1c485e0634fa17e8f9f253737823a876e4dd713ad715f11459072f48"} Jul 14 20:15:54.079527 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:54.079399 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3d4313c1c485e0634fa17e8f9f253737823a876e4dd713ad715f11459072f48" Jul 14 20:15:54.117426 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:54.117388 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx"] Jul 14 20:15:54.134356 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:54.134324 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-z7xrx"] Jul 14 20:15:54.883714 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:54.883679 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" path="/var/lib/kubelet/pods/d776c3eb-91f7-46bb-b499-e4a5322bd042/volumes" Jul 14 20:15:58.944981 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:58.944952 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:15:58.949489 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:58.949463 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:15:58.951111 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:58.951088 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:15:58.955557 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:15:58.955538 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:16:26.530502 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.530463 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n"] Jul 14 20:16:26.530971 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.530890 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:16:26.530971 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.530901 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:16:26.530971 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.530926 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:16:26.530971 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.530932 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:16:26.531109 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.530991 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:16:26.531109 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.531001 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:16:26.534203 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.534186 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" Jul 14 20:16:26.536910 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.536883 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:16:26.537044 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.536931 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:16:26.542319 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.542196 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n"] Jul 14 20:16:26.678012 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.677971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ada76-32c3-4383-86cd-7f7bd7614dcf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n\" (UID: \"349ada76-32c3-4383-86cd-7f7bd7614dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" Jul 14 20:16:26.779590 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.779552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ada76-32c3-4383-86cd-7f7bd7614dcf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n\" (UID: \"349ada76-32c3-4383-86cd-7f7bd7614dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" Jul 14 20:16:26.780226 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.780206 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ada76-32c3-4383-86cd-7f7bd7614dcf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n\" (UID: \"349ada76-32c3-4383-86cd-7f7bd7614dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" Jul 14 20:16:26.845794 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.845749 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" Jul 14 20:16:26.984088 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:26.984056 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n"] Jul 14 20:16:26.986200 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:16:26.986175 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349ada76_32c3_4383_86cd_7f7bd7614dcf.slice/crio-fa66d371998073b2fe09a86aa83dfc0bf43f1eb5b27fcaab95814af35152aac8 WatchSource:0}: Error finding container fa66d371998073b2fe09a86aa83dfc0bf43f1eb5b27fcaab95814af35152aac8: Status 404 returned error can't find the container with id fa66d371998073b2fe09a86aa83dfc0bf43f1eb5b27fcaab95814af35152aac8 Jul 14 20:16:27.260822 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:27.260786 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" event={"ID":"349ada76-32c3-4383-86cd-7f7bd7614dcf","Type":"ContainerStarted","Data":"fb99f3f7476272292c19c46d4d3f91ef5eb36d801eeffbd65716083988dcb89a"} Jul 14 20:16:27.260993 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:27.260879 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" event={"ID":"349ada76-32c3-4383-86cd-7f7bd7614dcf","Type":"ContainerStarted","Data":"fa66d371998073b2fe09a86aa83dfc0bf43f1eb5b27fcaab95814af35152aac8"} Jul 14 20:16:27.278442 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:27.278389 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" podStartSLOduration=1.278372808 podStartE2EDuration="1.278372808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:16:27.276014674 +0000 UTC m=+1828.923578471" watchObservedRunningTime="2026-07-14 20:16:27.278372808 +0000 UTC m=+1828.925936422" Jul 14 20:16:28.267792 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:28.267750 2578 generic.go:356] "Generic (PLEG): container finished" podID="349ada76-32c3-4383-86cd-7f7bd7614dcf" containerID="fb99f3f7476272292c19c46d4d3f91ef5eb36d801eeffbd65716083988dcb89a" exitCode=0 Jul 14 20:16:28.268211 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:28.267803 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" event={"ID":"349ada76-32c3-4383-86cd-7f7bd7614dcf","Type":"ContainerDied","Data":"fb99f3f7476272292c19c46d4d3f91ef5eb36d801eeffbd65716083988dcb89a"} Jul 14 20:16:29.420293 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:29.420263 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" Jul 14 20:16:29.506031 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:29.505985 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ada76-32c3-4383-86cd-7f7bd7614dcf-maas-api-service-ca\") pod \"349ada76-32c3-4383-86cd-7f7bd7614dcf\" (UID: \"349ada76-32c3-4383-86cd-7f7bd7614dcf\") " Jul 14 20:16:29.506384 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:29.506360 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349ada76-32c3-4383-86cd-7f7bd7614dcf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "349ada76-32c3-4383-86cd-7f7bd7614dcf" (UID: "349ada76-32c3-4383-86cd-7f7bd7614dcf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:16:29.607321 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:29.607280 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ada76-32c3-4383-86cd-7f7bd7614dcf-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:16:30.280591 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:30.280559 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" Jul 14 20:16:30.280591 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:30.280587 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n" event={"ID":"349ada76-32c3-4383-86cd-7f7bd7614dcf","Type":"ContainerDied","Data":"fa66d371998073b2fe09a86aa83dfc0bf43f1eb5b27fcaab95814af35152aac8"} Jul 14 20:16:30.280812 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:16:30.280639 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa66d371998073b2fe09a86aa83dfc0bf43f1eb5b27fcaab95814af35152aac8" Jul 14 20:18:44.708099 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.708062 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct"] Jul 14 20:18:44.708551 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.708475 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:18:44.708551 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.708487 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:18:44.708551 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.708498 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="349ada76-32c3-4383-86cd-7f7bd7614dcf" containerName="revoke-keys" Jul 14 20:18:44.708551 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.708504 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="349ada76-32c3-4383-86cd-7f7bd7614dcf" containerName="revoke-keys" Jul 14 20:18:44.708686 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.708574 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="349ada76-32c3-4383-86cd-7f7bd7614dcf" containerName="revoke-keys" Jul 14 20:18:44.708686 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.708584 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d776c3eb-91f7-46bb-b499-e4a5322bd042" containerName="revoke-keys" Jul 14 20:18:44.711651 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.711632 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" Jul 14 20:18:44.714810 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.714784 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:18:44.715255 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.715235 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:18:44.726060 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.726033 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct"] Jul 14 20:18:44.855235 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.855193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4137873f-2cc5-42a3-9907-ab980589017c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct\" (UID: \"4137873f-2cc5-42a3-9907-ab980589017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" Jul 14 20:18:44.956124 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.956079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4137873f-2cc5-42a3-9907-ab980589017c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct\" (UID: \"4137873f-2cc5-42a3-9907-ab980589017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" Jul 14 20:18:44.956761 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:44.956738 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4137873f-2cc5-42a3-9907-ab980589017c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct\" (UID: \"4137873f-2cc5-42a3-9907-ab980589017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" Jul 14 20:18:45.022686 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:45.022583 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" Jul 14 20:18:45.162280 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:45.162246 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct"] Jul 14 20:18:45.163193 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:18:45.163165 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4137873f_2cc5_42a3_9907_ab980589017c.slice/crio-e3b6c944dbd187ddd8d33f3ff9b5033091285e97f095d880fc039b2ee6594454 WatchSource:0}: Error finding container e3b6c944dbd187ddd8d33f3ff9b5033091285e97f095d880fc039b2ee6594454: Status 404 returned error can't find the container with id e3b6c944dbd187ddd8d33f3ff9b5033091285e97f095d880fc039b2ee6594454 Jul 14 20:18:46.014296 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:46.014265 2578 generic.go:356] "Generic (PLEG): container finished" podID="4137873f-2cc5-42a3-9907-ab980589017c" containerID="3bc42aedade728b2c3c00d3ad56c3cd941815044735b76d891fcdce075a91426" exitCode=0 Jul 14 20:18:46.014655 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:46.014345 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" event={"ID":"4137873f-2cc5-42a3-9907-ab980589017c","Type":"ContainerDied","Data":"3bc42aedade728b2c3c00d3ad56c3cd941815044735b76d891fcdce075a91426"} Jul 14 20:18:46.014655 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:46.014397 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" event={"ID":"4137873f-2cc5-42a3-9907-ab980589017c","Type":"ContainerStarted","Data":"e3b6c944dbd187ddd8d33f3ff9b5033091285e97f095d880fc039b2ee6594454"} Jul 14 20:18:47.164287 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:47.164259 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" Jul 14 20:18:47.279961 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:47.276283 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4137873f-2cc5-42a3-9907-ab980589017c-maas-api-service-ca\") pod \"4137873f-2cc5-42a3-9907-ab980589017c\" (UID: \"4137873f-2cc5-42a3-9907-ab980589017c\") " Jul 14 20:18:47.279961 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:47.277226 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4137873f-2cc5-42a3-9907-ab980589017c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4137873f-2cc5-42a3-9907-ab980589017c" (UID: "4137873f-2cc5-42a3-9907-ab980589017c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:18:47.377431 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:47.377393 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4137873f-2cc5-42a3-9907-ab980589017c-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:18:48.026456 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:48.026415 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" event={"ID":"4137873f-2cc5-42a3-9907-ab980589017c","Type":"ContainerDied","Data":"e3b6c944dbd187ddd8d33f3ff9b5033091285e97f095d880fc039b2ee6594454"} Jul 14 20:18:48.026658 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:48.026473 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3b6c944dbd187ddd8d33f3ff9b5033091285e97f095d880fc039b2ee6594454" Jul 14 20:18:48.026658 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:18:48.026425 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct" Jul 14 20:20:16.760565 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.760526 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq"] Jul 14 20:20:16.761106 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.760972 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4137873f-2cc5-42a3-9907-ab980589017c" containerName="revoke-keys" Jul 14 20:20:16.761106 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.760986 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4137873f-2cc5-42a3-9907-ab980589017c" containerName="revoke-keys" Jul 14 20:20:16.761106 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.761072 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4137873f-2cc5-42a3-9907-ab980589017c" containerName="revoke-keys" Jul 14 20:20:16.763951 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.763932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" Jul 14 20:20:16.767952 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.767927 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:20:16.768126 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.768105 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:20:16.774869 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.774845 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq"] Jul 14 20:20:16.798819 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.798794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64144d3-a7f2-4800-8e03-74354b3cdbd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq\" (UID: \"e64144d3-a7f2-4800-8e03-74354b3cdbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" Jul 14 20:20:16.899799 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.899764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64144d3-a7f2-4800-8e03-74354b3cdbd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq\" (UID: \"e64144d3-a7f2-4800-8e03-74354b3cdbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" Jul 14 20:20:16.900423 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:16.900388 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64144d3-a7f2-4800-8e03-74354b3cdbd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq\" (UID: \"e64144d3-a7f2-4800-8e03-74354b3cdbd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" Jul 14 20:20:17.075360 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:17.075322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" Jul 14 20:20:17.216776 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:17.216750 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq"] Jul 14 20:20:17.219066 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:20:17.219037 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode64144d3_a7f2_4800_8e03_74354b3cdbd2.slice/crio-cae821b57f765f7178bb0d4e57c61fc7a4557951767a4a8c42d2706874da9f78 WatchSource:0}: Error finding container cae821b57f765f7178bb0d4e57c61fc7a4557951767a4a8c42d2706874da9f78: Status 404 returned error can't find the container with id cae821b57f765f7178bb0d4e57c61fc7a4557951767a4a8c42d2706874da9f78 Jul 14 20:20:17.511190 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:17.511150 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" event={"ID":"e64144d3-a7f2-4800-8e03-74354b3cdbd2","Type":"ContainerStarted","Data":"caaf980d0ee4ae02ef3867df44f499ae4da8280ef7c4e35107935c0258a2671f"} Jul 14 20:20:17.511387 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:17.511237 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" event={"ID":"e64144d3-a7f2-4800-8e03-74354b3cdbd2","Type":"ContainerStarted","Data":"cae821b57f765f7178bb0d4e57c61fc7a4557951767a4a8c42d2706874da9f78"} Jul 14 20:20:17.531089 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:17.531038 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" podStartSLOduration=1.531023526 podStartE2EDuration="1.531023526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:20:17.52899356 +0000 UTC m=+2059.176557126" watchObservedRunningTime="2026-07-14 20:20:17.531023526 +0000 UTC m=+2059.178587090" Jul 14 20:20:18.517840 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:18.517800 2578 generic.go:356] "Generic (PLEG): container finished" podID="e64144d3-a7f2-4800-8e03-74354b3cdbd2" containerID="caaf980d0ee4ae02ef3867df44f499ae4da8280ef7c4e35107935c0258a2671f" exitCode=0 Jul 14 20:20:18.518258 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:18.517877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" event={"ID":"e64144d3-a7f2-4800-8e03-74354b3cdbd2","Type":"ContainerDied","Data":"caaf980d0ee4ae02ef3867df44f499ae4da8280ef7c4e35107935c0258a2671f"} Jul 14 20:20:19.669378 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:19.669352 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" Jul 14 20:20:19.729474 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:19.729439 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64144d3-a7f2-4800-8e03-74354b3cdbd2-maas-api-service-ca\") pod \"e64144d3-a7f2-4800-8e03-74354b3cdbd2\" (UID: \"e64144d3-a7f2-4800-8e03-74354b3cdbd2\") " Jul 14 20:20:19.729780 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:19.729757 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64144d3-a7f2-4800-8e03-74354b3cdbd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e64144d3-a7f2-4800-8e03-74354b3cdbd2" (UID: "e64144d3-a7f2-4800-8e03-74354b3cdbd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:20:19.831201 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:19.831164 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e64144d3-a7f2-4800-8e03-74354b3cdbd2-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:20:20.530666 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:20.530639 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" Jul 14 20:20:20.530863 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:20.530638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq" event={"ID":"e64144d3-a7f2-4800-8e03-74354b3cdbd2","Type":"ContainerDied","Data":"cae821b57f765f7178bb0d4e57c61fc7a4557951767a4a8c42d2706874da9f78"} Jul 14 20:20:20.530863 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:20.530758 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cae821b57f765f7178bb0d4e57c61fc7a4557951767a4a8c42d2706874da9f78" Jul 14 20:20:58.978451 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:58.978419 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:20:58.984044 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:58.984020 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:20:58.984852 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:58.984809 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:20:58.990082 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:58.990061 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:20:59.059007 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:20:59.058974 2578 scope.go:117] "RemoveContainer" containerID="77fbcb870155bc52c2a9831b4adabbe5d3ec57d21532746a56ebeb6d045175d6" Jul 14 20:21:59.079872 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:21:59.079782 2578 scope.go:117] "RemoveContainer" containerID="d915422b5e615e03c1509b8fb7b5683c4b58d16285eff1c311096b9aebe4b2b8" Jul 14 20:21:59.088621 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:21:59.088602 2578 scope.go:117] "RemoveContainer" containerID="3b73619b995bf3f3ed85d57ee06713470b875f37124339f379e891abd98981f5" Jul 14 20:23:49.249604 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.249566 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc"] Jul 14 20:23:49.252057 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.249993 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e64144d3-a7f2-4800-8e03-74354b3cdbd2" containerName="revoke-keys" Jul 14 20:23:49.252057 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.250006 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e64144d3-a7f2-4800-8e03-74354b3cdbd2" containerName="revoke-keys" Jul 14 20:23:49.252057 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.250075 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e64144d3-a7f2-4800-8e03-74354b3cdbd2" containerName="revoke-keys" Jul 14 20:23:49.253055 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.253039 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" Jul 14 20:23:49.255482 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.255460 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:23:49.255482 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.255476 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:23:49.259756 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.259727 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc"] Jul 14 20:23:49.271386 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.271360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bf502-a8ed-4eb4-ae77-09d62fcd7f04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc\" (UID: \"f38bf502-a8ed-4eb4-ae77-09d62fcd7f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" Jul 14 20:23:49.372899 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.372861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bf502-a8ed-4eb4-ae77-09d62fcd7f04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc\" (UID: \"f38bf502-a8ed-4eb4-ae77-09d62fcd7f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" Jul 14 20:23:49.373671 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.373647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bf502-a8ed-4eb4-ae77-09d62fcd7f04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc\" (UID: \"f38bf502-a8ed-4eb4-ae77-09d62fcd7f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" Jul 14 20:23:49.564585 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.564556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" Jul 14 20:23:49.707271 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:49.707244 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc"] Jul 14 20:23:49.709311 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:23:49.709286 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38bf502_a8ed_4eb4_ae77_09d62fcd7f04.slice/crio-402ebdd3a07855b28c2450fdbdb4ee3cb5c31b0ab7545308f4c0f4cb0674dcb1 WatchSource:0}: Error finding container 402ebdd3a07855b28c2450fdbdb4ee3cb5c31b0ab7545308f4c0f4cb0674dcb1: Status 404 returned error can't find the container with id 402ebdd3a07855b28c2450fdbdb4ee3cb5c31b0ab7545308f4c0f4cb0674dcb1 Jul 14 20:23:50.670824 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:50.670786 2578 generic.go:356] "Generic (PLEG): container finished" podID="f38bf502-a8ed-4eb4-ae77-09d62fcd7f04" containerID="68d4507b76d89360d867d09e861499b2bfd4b000e239ddd4a8ff257656e94bbc" exitCode=0 Jul 14 20:23:50.671290 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:50.670871 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" event={"ID":"f38bf502-a8ed-4eb4-ae77-09d62fcd7f04","Type":"ContainerDied","Data":"68d4507b76d89360d867d09e861499b2bfd4b000e239ddd4a8ff257656e94bbc"} Jul 14 20:23:50.671290 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:50.670946 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" event={"ID":"f38bf502-a8ed-4eb4-ae77-09d62fcd7f04","Type":"ContainerStarted","Data":"402ebdd3a07855b28c2450fdbdb4ee3cb5c31b0ab7545308f4c0f4cb0674dcb1"} Jul 14 20:23:51.824950 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:51.824923 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" Jul 14 20:23:51.898574 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:51.898539 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bf502-a8ed-4eb4-ae77-09d62fcd7f04-maas-api-service-ca\") pod \"f38bf502-a8ed-4eb4-ae77-09d62fcd7f04\" (UID: \"f38bf502-a8ed-4eb4-ae77-09d62fcd7f04\") " Jul 14 20:23:51.898949 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:51.898928 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38bf502-a8ed-4eb4-ae77-09d62fcd7f04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f38bf502-a8ed-4eb4-ae77-09d62fcd7f04" (UID: "f38bf502-a8ed-4eb4-ae77-09d62fcd7f04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:23:51.999406 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:51.999320 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bf502-a8ed-4eb4-ae77-09d62fcd7f04-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:23:52.683123 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:52.683090 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" Jul 14 20:23:52.683305 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:52.683092 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc" event={"ID":"f38bf502-a8ed-4eb4-ae77-09d62fcd7f04","Type":"ContainerDied","Data":"402ebdd3a07855b28c2450fdbdb4ee3cb5c31b0ab7545308f4c0f4cb0674dcb1"} Jul 14 20:23:52.683305 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:23:52.683217 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="402ebdd3a07855b28c2450fdbdb4ee3cb5c31b0ab7545308f4c0f4cb0674dcb1" Jul 14 20:24:15.406722 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.406682 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6"] Jul 14 20:24:15.407341 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.407327 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f38bf502-a8ed-4eb4-ae77-09d62fcd7f04" containerName="revoke-keys" Jul 14 20:24:15.407416 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.407347 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f38bf502-a8ed-4eb4-ae77-09d62fcd7f04" containerName="revoke-keys" Jul 14 20:24:15.407503 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.407488 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f38bf502-a8ed-4eb4-ae77-09d62fcd7f04" containerName="revoke-keys" Jul 14 20:24:15.410649 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.410628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" Jul 14 20:24:15.413230 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.413204 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:24:15.413351 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.413239 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:24:15.417342 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.417322 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6"] Jul 14 20:24:15.519623 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.519589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c112c97-6433-4578-8d68-b4863cdc2b50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6\" (UID: \"5c112c97-6433-4578-8d68-b4863cdc2b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" Jul 14 20:24:15.620951 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.620909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c112c97-6433-4578-8d68-b4863cdc2b50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6\" (UID: \"5c112c97-6433-4578-8d68-b4863cdc2b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" Jul 14 20:24:15.621573 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.621551 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c112c97-6433-4578-8d68-b4863cdc2b50-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6\" (UID: \"5c112c97-6433-4578-8d68-b4863cdc2b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" Jul 14 20:24:15.723264 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.723173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" Jul 14 20:24:15.872837 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:15.872797 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6"] Jul 14 20:24:15.874367 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:24:15.874342 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c112c97_6433_4578_8d68_b4863cdc2b50.slice/crio-12fd2b6845ceefda8c449cbe14589b3a80713862d4549dc7f6703678e4245b23 WatchSource:0}: Error finding container 12fd2b6845ceefda8c449cbe14589b3a80713862d4549dc7f6703678e4245b23: Status 404 returned error can't find the container with id 12fd2b6845ceefda8c449cbe14589b3a80713862d4549dc7f6703678e4245b23 Jul 14 20:24:16.817413 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:16.817377 2578 generic.go:356] "Generic (PLEG): container finished" podID="5c112c97-6433-4578-8d68-b4863cdc2b50" containerID="635782b2a8caeb59cd67b901444b91098ff5177aa09e0d460ae7be291bc97dc5" exitCode=0 Jul 14 20:24:16.817866 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:16.817453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" event={"ID":"5c112c97-6433-4578-8d68-b4863cdc2b50","Type":"ContainerDied","Data":"635782b2a8caeb59cd67b901444b91098ff5177aa09e0d460ae7be291bc97dc5"} Jul 14 20:24:16.817866 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:16.817505 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" event={"ID":"5c112c97-6433-4578-8d68-b4863cdc2b50","Type":"ContainerStarted","Data":"12fd2b6845ceefda8c449cbe14589b3a80713862d4549dc7f6703678e4245b23"} Jul 14 20:24:17.965616 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:17.965587 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" Jul 14 20:24:18.044742 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:18.044708 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c112c97-6433-4578-8d68-b4863cdc2b50-maas-api-service-ca\") pod \"5c112c97-6433-4578-8d68-b4863cdc2b50\" (UID: \"5c112c97-6433-4578-8d68-b4863cdc2b50\") " Jul 14 20:24:18.045167 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:18.045141 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c112c97-6433-4578-8d68-b4863cdc2b50-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c112c97-6433-4578-8d68-b4863cdc2b50" (UID: "5c112c97-6433-4578-8d68-b4863cdc2b50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:24:18.145783 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:18.145705 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c112c97-6433-4578-8d68-b4863cdc2b50-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:24:18.829110 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:18.829072 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" event={"ID":"5c112c97-6433-4578-8d68-b4863cdc2b50","Type":"ContainerDied","Data":"12fd2b6845ceefda8c449cbe14589b3a80713862d4549dc7f6703678e4245b23"} Jul 14 20:24:18.829110 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:18.829092 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6" Jul 14 20:24:18.829365 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:24:18.829121 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12fd2b6845ceefda8c449cbe14589b3a80713862d4549dc7f6703678e4245b23" Jul 14 20:25:05.449746 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.449618 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq"] Jul 14 20:25:05.450400 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.450360 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c112c97-6433-4578-8d68-b4863cdc2b50" containerName="revoke-keys" Jul 14 20:25:05.450400 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.450384 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c112c97-6433-4578-8d68-b4863cdc2b50" containerName="revoke-keys" Jul 14 20:25:05.450522 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.450479 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c112c97-6433-4578-8d68-b4863cdc2b50" containerName="revoke-keys" Jul 14 20:25:05.453616 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.453592 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" Jul 14 20:25:05.456278 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.456247 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:25:05.456482 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.456317 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:25:05.461399 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.461367 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq"] Jul 14 20:25:05.584556 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.584505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed018c8-e271-48c8-be16-28a5f8678b6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-285387-42pqq\" (UID: \"6ed018c8-e271-48c8-be16-28a5f8678b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" Jul 14 20:25:05.685617 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.685571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed018c8-e271-48c8-be16-28a5f8678b6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-285387-42pqq\" (UID: \"6ed018c8-e271-48c8-be16-28a5f8678b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" Jul 14 20:25:05.686279 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.686258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed018c8-e271-48c8-be16-28a5f8678b6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-285387-42pqq\" (UID: \"6ed018c8-e271-48c8-be16-28a5f8678b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" Jul 14 20:25:05.766903 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.766798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" Jul 14 20:25:05.910859 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:05.910806 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq"] Jul 14 20:25:05.912623 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:25:05.912594 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed018c8_e271_48c8_be16_28a5f8678b6a.slice/crio-fb4159d887959bd26f02f85204dc8fe39672de65505233c1e0b54e2b67b5ffb7 WatchSource:0}: Error finding container fb4159d887959bd26f02f85204dc8fe39672de65505233c1e0b54e2b67b5ffb7: Status 404 returned error can't find the container with id fb4159d887959bd26f02f85204dc8fe39672de65505233c1e0b54e2b67b5ffb7 Jul 14 20:25:06.103257 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:06.103217 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" event={"ID":"6ed018c8-e271-48c8-be16-28a5f8678b6a","Type":"ContainerStarted","Data":"fb4159d887959bd26f02f85204dc8fe39672de65505233c1e0b54e2b67b5ffb7"} Jul 14 20:25:07.109917 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:07.109885 2578 generic.go:356] "Generic (PLEG): container finished" podID="6ed018c8-e271-48c8-be16-28a5f8678b6a" containerID="70fa1153b97e5485da583cd0f7fd828a5351871d6923055db510816e19ffb91b" exitCode=0 Jul 14 20:25:07.110307 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:07.109977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" event={"ID":"6ed018c8-e271-48c8-be16-28a5f8678b6a","Type":"ContainerDied","Data":"70fa1153b97e5485da583cd0f7fd828a5351871d6923055db510816e19ffb91b"} Jul 14 20:25:08.260634 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:08.260601 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" Jul 14 20:25:08.415767 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:08.415688 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed018c8-e271-48c8-be16-28a5f8678b6a-maas-api-service-ca\") pod \"6ed018c8-e271-48c8-be16-28a5f8678b6a\" (UID: \"6ed018c8-e271-48c8-be16-28a5f8678b6a\") " Jul 14 20:25:08.416041 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:08.416020 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed018c8-e271-48c8-be16-28a5f8678b6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ed018c8-e271-48c8-be16-28a5f8678b6a" (UID: "6ed018c8-e271-48c8-be16-28a5f8678b6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:25:08.517139 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:08.517103 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ed018c8-e271-48c8-be16-28a5f8678b6a-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:25:09.122503 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:09.122467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" event={"ID":"6ed018c8-e271-48c8-be16-28a5f8678b6a","Type":"ContainerDied","Data":"fb4159d887959bd26f02f85204dc8fe39672de65505233c1e0b54e2b67b5ffb7"} Jul 14 20:25:09.122674 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:09.122508 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-285387-42pqq" Jul 14 20:25:09.122674 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:09.122516 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb4159d887959bd26f02f85204dc8fe39672de65505233c1e0b54e2b67b5ffb7" Jul 14 20:25:31.566620 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.566577 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75"] Jul 14 20:25:31.567091 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.567076 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed018c8-e271-48c8-be16-28a5f8678b6a" containerName="revoke-keys" Jul 14 20:25:31.567091 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.567093 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed018c8-e271-48c8-be16-28a5f8678b6a" containerName="revoke-keys" Jul 14 20:25:31.567182 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.567172 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed018c8-e271-48c8-be16-28a5f8678b6a" containerName="revoke-keys" Jul 14 20:25:31.570556 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.570537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" Jul 14 20:25:31.573191 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.573161 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:25:31.573774 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.573748 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:25:31.576542 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.576510 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75"] Jul 14 20:25:31.612722 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.612683 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f96987-ef73-4641-bef9-7a98f8a5858c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75\" (UID: \"b4f96987-ef73-4641-bef9-7a98f8a5858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" Jul 14 20:25:31.714231 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.714184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f96987-ef73-4641-bef9-7a98f8a5858c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75\" (UID: \"b4f96987-ef73-4641-bef9-7a98f8a5858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" Jul 14 20:25:31.714901 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.714881 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f96987-ef73-4641-bef9-7a98f8a5858c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75\" (UID: \"b4f96987-ef73-4641-bef9-7a98f8a5858c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" Jul 14 20:25:31.882982 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:31.882887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" Jul 14 20:25:32.021721 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:32.021691 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75"] Jul 14 20:25:32.023696 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:25:32.023663 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f96987_ef73_4641_bef9_7a98f8a5858c.slice/crio-0dfcb602a63c64266cee38e889d0d78c4d28aa6e5fd7d48a141a62ee5840ae5c WatchSource:0}: Error finding container 0dfcb602a63c64266cee38e889d0d78c4d28aa6e5fd7d48a141a62ee5840ae5c: Status 404 returned error can't find the container with id 0dfcb602a63c64266cee38e889d0d78c4d28aa6e5fd7d48a141a62ee5840ae5c Jul 14 20:25:32.251517 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:32.251477 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" event={"ID":"b4f96987-ef73-4641-bef9-7a98f8a5858c","Type":"ContainerStarted","Data":"0dfcb602a63c64266cee38e889d0d78c4d28aa6e5fd7d48a141a62ee5840ae5c"} Jul 14 20:25:33.259837 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:33.259726 2578 generic.go:356] "Generic (PLEG): container finished" podID="b4f96987-ef73-4641-bef9-7a98f8a5858c" containerID="9692c533a844ffc87aaa8d802528c6f6f0a409e047f5cf38e08d80594bdfb19d" exitCode=0 Jul 14 20:25:33.260213 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:33.259816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" event={"ID":"b4f96987-ef73-4641-bef9-7a98f8a5858c","Type":"ContainerDied","Data":"9692c533a844ffc87aaa8d802528c6f6f0a409e047f5cf38e08d80594bdfb19d"} Jul 14 20:25:34.407246 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:34.407220 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" Jul 14 20:25:34.441018 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:34.440984 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f96987-ef73-4641-bef9-7a98f8a5858c-maas-api-service-ca\") pod \"b4f96987-ef73-4641-bef9-7a98f8a5858c\" (UID: \"b4f96987-ef73-4641-bef9-7a98f8a5858c\") " Jul 14 20:25:34.441435 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:34.441404 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f96987-ef73-4641-bef9-7a98f8a5858c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4f96987-ef73-4641-bef9-7a98f8a5858c" (UID: "b4f96987-ef73-4641-bef9-7a98f8a5858c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:25:34.542059 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:34.541967 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f96987-ef73-4641-bef9-7a98f8a5858c-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:25:35.271387 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:35.271296 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" Jul 14 20:25:35.271387 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:35.271298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75" event={"ID":"b4f96987-ef73-4641-bef9-7a98f8a5858c","Type":"ContainerDied","Data":"0dfcb602a63c64266cee38e889d0d78c4d28aa6e5fd7d48a141a62ee5840ae5c"} Jul 14 20:25:35.271622 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:35.271425 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dfcb602a63c64266cee38e889d0d78c4d28aa6e5fd7d48a141a62ee5840ae5c" Jul 14 20:25:59.012873 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:59.012844 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:25:59.019362 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:59.019334 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:25:59.020345 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:59.020321 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:25:59.029789 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:25:59.029763 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:26:53.081737 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.081698 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb"] Jul 14 20:26:53.082388 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.082369 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f96987-ef73-4641-bef9-7a98f8a5858c" containerName="revoke-keys" Jul 14 20:26:53.082437 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.082391 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f96987-ef73-4641-bef9-7a98f8a5858c" containerName="revoke-keys" Jul 14 20:26:53.082511 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.082492 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f96987-ef73-4641-bef9-7a98f8a5858c" containerName="revoke-keys" Jul 14 20:26:53.086209 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.086186 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.088817 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.088793 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"auth-test-model-a99cbd-kserve-self-signed-certs\"" Jul 14 20:26:53.088817 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.088807 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"openshift-service-ca.crt\"" Jul 14 20:26:53.090124 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.090074 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"kube-root-ca.crt\"" Jul 14 20:26:53.091011 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.090344 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a99cbd\"/\"default-dockercfg-5fhmn\"" Jul 14 20:26:53.093069 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.093047 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb"] Jul 14 20:26:53.171487 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.171443 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-model-cache\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.171697 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.171512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7lb6\" (UniqueName: \"kubernetes.io/projected/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-kube-api-access-p7lb6\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.171697 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.171558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tls-certs\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.171697 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.171619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tmp-dir\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.171697 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.171658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-dshm\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.171939 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.171736 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-home\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272354 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tmp-dir\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272540 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-dshm\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272540 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-home\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272540 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-model-cache\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272540 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7lb6\" (UniqueName: \"kubernetes.io/projected/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-kube-api-access-p7lb6\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272772 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tls-certs\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272855 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tmp-dir\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272855 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272846 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-home\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.272977 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.272930 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-model-cache\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.274758 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.274726 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-dshm\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.275489 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.275470 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tls-certs\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.283043 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.283011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7lb6\" (UniqueName: \"kubernetes.io/projected/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-kube-api-access-p7lb6\") pod \"auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.399377 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.399283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:26:53.541758 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.541724 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb"] Jul 14 20:26:53.542994 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:26:53.542965 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod716f52ed_18f7_4db4_bba8_f34eb3ca4b1d.slice/crio-e3bff4035499fe240b4cb96c39b59f749b93bd53220171ebeccb231825c43145 WatchSource:0}: Error finding container e3bff4035499fe240b4cb96c39b59f749b93bd53220171ebeccb231825c43145: Status 404 returned error can't find the container with id e3bff4035499fe240b4cb96c39b59f749b93bd53220171ebeccb231825c43145 Jul 14 20:26:53.711264 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.711152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" event={"ID":"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d","Type":"ContainerStarted","Data":"628a54aa1865e82687569baddf7b57bb92f288e1fd74e92cb047597859a100c7"} Jul 14 20:26:53.711264 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.711220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" event={"ID":"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d","Type":"ContainerStarted","Data":"e3bff4035499fe240b4cb96c39b59f749b93bd53220171ebeccb231825c43145"} Jul 14 20:26:53.728936 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:26:53.728887 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" podStartSLOduration=0.728873354 podStartE2EDuration="728.873354ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:26:53.726496964 +0000 UTC m=+2455.374060560" watchObservedRunningTime="2026-07-14 20:26:53.728873354 +0000 UTC m=+2455.376436921" Jul 14 20:27:03.399467 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:27:03.399423 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:27:03.399905 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:27:03.399485 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:27:03.412312 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:27:03.412284 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:27:03.782858 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:27:03.782749 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:29:06.280471 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:06.280425 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb"] Jul 14 20:29:21.302430 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.302331 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" podUID="716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" containerName="main" containerID="cri-o://628a54aa1865e82687569baddf7b57bb92f288e1fd74e92cb047597859a100c7" gracePeriod=45 Jul 14 20:29:21.554679 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.554605 2578 generic.go:356] "Generic (PLEG): container finished" podID="716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" containerID="628a54aa1865e82687569baddf7b57bb92f288e1fd74e92cb047597859a100c7" exitCode=0 Jul 14 20:29:21.554679 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.554661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" event={"ID":"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d","Type":"ContainerDied","Data":"628a54aa1865e82687569baddf7b57bb92f288e1fd74e92cb047597859a100c7"} Jul 14 20:29:21.575587 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.575560 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:29:21.761531 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.761490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7lb6\" (UniqueName: \"kubernetes.io/projected/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-kube-api-access-p7lb6\") pod \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " Jul 14 20:29:21.761732 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.761584 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-home\") pod \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " Jul 14 20:29:21.761732 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.761621 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-dshm\") pod \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " Jul 14 20:29:21.761732 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.761640 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tmp-dir\") pod \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " Jul 14 20:29:21.761732 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.761665 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tls-certs\") pod \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " Jul 14 20:29:21.762011 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.761744 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-model-cache\") pod \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\" (UID: \"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d\") " Jul 14 20:29:21.762011 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.761914 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-home" (OuterVolumeSpecName: "home") pod "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" (UID: "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:29:21.762011 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.761929 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" (UID: "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:29:21.762182 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.762160 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" (UID: "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:29:21.762182 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.762172 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tmp-dir\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:29:21.762291 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.762218 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-home\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:29:21.763974 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.763952 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" (UID: "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:29:21.764214 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.764187 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-dshm" (OuterVolumeSpecName: "dshm") pod "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" (UID: "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:29:21.764316 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.764218 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-kube-api-access-p7lb6" (OuterVolumeSpecName: "kube-api-access-p7lb6") pod "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" (UID: "716f52ed-18f7-4db4-bba8-f34eb3ca4b1d"). InnerVolumeSpecName "kube-api-access-p7lb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:29:21.863707 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.863609 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-tls-certs\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:29:21.863707 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.863641 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-model-cache\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:29:21.863707 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.863650 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7lb6\" (UniqueName: \"kubernetes.io/projected/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-kube-api-access-p7lb6\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:29:21.863707 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:21.863659 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d-dshm\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:29:22.562330 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:22.562301 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" Jul 14 20:29:22.562861 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:22.562340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb" event={"ID":"716f52ed-18f7-4db4-bba8-f34eb3ca4b1d","Type":"ContainerDied","Data":"e3bff4035499fe240b4cb96c39b59f749b93bd53220171ebeccb231825c43145"} Jul 14 20:29:22.562861 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:22.562395 2578 scope.go:117] "RemoveContainer" containerID="628a54aa1865e82687569baddf7b57bb92f288e1fd74e92cb047597859a100c7" Jul 14 20:29:22.586957 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:22.586917 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb"] Jul 14 20:29:22.591093 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:22.591063 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a99cbd/auth-test-model-a99cbd-kserve-6d7fb977c9-wxvjb"] Jul 14 20:29:22.884351 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:29:22.884264 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" path="/var/lib/kubelet/pods/716f52ed-18f7-4db4-bba8-f34eb3ca4b1d/volumes" Jul 14 20:30:00.143160 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.143110 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk"] Jul 14 20:30:00.143602 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.143558 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" containerName="main" Jul 14 20:30:00.143602 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.143572 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" containerName="main" Jul 14 20:30:00.143681 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.143644 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="716f52ed-18f7-4db4-bba8-f34eb3ca4b1d" containerName="main" Jul 14 20:30:00.147986 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.147966 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84"] Jul 14 20:30:00.148107 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.148085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" Jul 14 20:30:00.150628 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.150603 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:30:00.151523 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.151505 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n"] Jul 14 20:30:00.151627 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.151610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" Jul 14 20:30:00.154961 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.154937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n" Jul 14 20:30:00.159475 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.159446 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk"] Jul 14 20:30:00.161013 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.160990 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" Jul 14 20:30:00.162811 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.162783 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n"] Jul 14 20:30:00.166541 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.166519 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84"] Jul 14 20:30:00.170889 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.170866 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" Jul 14 20:30:00.210377 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.210344 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n" Jul 14 20:30:00.352685 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.352579 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk"] Jul 14 20:30:00.355603 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:30:00.355570 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2d1044_2625_4a5d_a72e_8ebab927a3bc.slice/crio-dbf64a6196756d498d80c5cffdc8568f59440cf1bfbd561451e1a725aab4a2f6 WatchSource:0}: Error finding container dbf64a6196756d498d80c5cffdc8568f59440cf1bfbd561451e1a725aab4a2f6: Status 404 returned error can't find the container with id dbf64a6196756d498d80c5cffdc8568f59440cf1bfbd561451e1a725aab4a2f6 Jul 14 20:30:00.411596 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.411570 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n"] Jul 14 20:30:00.413209 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:30:00.413175 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d662aee_601e_4faf_aa07_9629f8834ab9.slice/crio-0f15daf27676ecf6d5c27ad3a000b11e1ab0599891be99ff42a39b29582269fc WatchSource:0}: Error finding container 0f15daf27676ecf6d5c27ad3a000b11e1ab0599891be99ff42a39b29582269fc: Status 404 returned error can't find the container with id 0f15daf27676ecf6d5c27ad3a000b11e1ab0599891be99ff42a39b29582269fc Jul 14 20:30:00.597674 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.597649 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84"] Jul 14 20:30:00.599575 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:30:00.599539 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77c3031_3b4e_4566_86ea_3b6f2db61c01.slice/crio-748d3a30d8f2ef9cacce5407c7c61f371f6d3d0ecedae763b72db8e0e99d2f80 WatchSource:0}: Error finding container 748d3a30d8f2ef9cacce5407c7c61f371f6d3d0ecedae763b72db8e0e99d2f80: Status 404 returned error can't find the container with id 748d3a30d8f2ef9cacce5407c7c61f371f6d3d0ecedae763b72db8e0e99d2f80 Jul 14 20:30:00.785817 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.785777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" event={"ID":"0a2d1044-2625-4a5d-a72e-8ebab927a3bc","Type":"ContainerStarted","Data":"efe8569942ef7bc2b02c3309bd6b1fb5ce2c8dc1eb606b67caba5f991eae7b01"} Jul 14 20:30:00.786033 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.785862 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" event={"ID":"0a2d1044-2625-4a5d-a72e-8ebab927a3bc","Type":"ContainerStarted","Data":"dbf64a6196756d498d80c5cffdc8568f59440cf1bfbd561451e1a725aab4a2f6"} Jul 14 20:30:00.786961 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.786932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n" event={"ID":"5d662aee-601e-4faf-aa07-9629f8834ab9","Type":"ContainerStarted","Data":"0f15daf27676ecf6d5c27ad3a000b11e1ab0599891be99ff42a39b29582269fc"} Jul 14 20:30:00.788085 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.788059 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" event={"ID":"b77c3031-3b4e-4566-86ea-3b6f2db61c01","Type":"ContainerStarted","Data":"748d3a30d8f2ef9cacce5407c7c61f371f6d3d0ecedae763b72db8e0e99d2f80"} Jul 14 20:30:00.802770 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:00.802717 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" podStartSLOduration=0.802699698 podStartE2EDuration="802.699698ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:30:00.802052646 +0000 UTC m=+2642.449616213" watchObservedRunningTime="2026-07-14 20:30:00.802699698 +0000 UTC m=+2642.450263263" Jul 14 20:30:01.795523 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:01.795423 2578 generic.go:356] "Generic (PLEG): container finished" podID="0a2d1044-2625-4a5d-a72e-8ebab927a3bc" containerID="efe8569942ef7bc2b02c3309bd6b1fb5ce2c8dc1eb606b67caba5f991eae7b01" exitCode=0 Jul 14 20:30:01.795523 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:01.795509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" event={"ID":"0a2d1044-2625-4a5d-a72e-8ebab927a3bc","Type":"ContainerDied","Data":"efe8569942ef7bc2b02c3309bd6b1fb5ce2c8dc1eb606b67caba5f991eae7b01"} Jul 14 20:30:01.797071 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:01.797046 2578 generic.go:356] "Generic (PLEG): container finished" podID="5d662aee-601e-4faf-aa07-9629f8834ab9" containerID="550c3e7faf0fc6aff1c50950ec89498be60f9768039ea5d851285af32973e61b" exitCode=0 Jul 14 20:30:01.797194 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:01.797112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n" event={"ID":"5d662aee-601e-4faf-aa07-9629f8834ab9","Type":"ContainerDied","Data":"550c3e7faf0fc6aff1c50950ec89498be60f9768039ea5d851285af32973e61b"} Jul 14 20:30:01.798484 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:01.798461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" event={"ID":"b77c3031-3b4e-4566-86ea-3b6f2db61c01","Type":"ContainerStarted","Data":"fd03089eaad52ab0000ef4f184bd7d68bbbe6e5069009c49abd33e0b52771582"} Jul 14 20:30:01.830225 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:01.830163 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" podStartSLOduration=1.8301456090000001 podStartE2EDuration="1.830145609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:30:01.827940133 +0000 UTC m=+2643.475503700" watchObservedRunningTime="2026-07-14 20:30:01.830145609 +0000 UTC m=+2643.477709175" Jul 14 20:30:02.805260 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:02.805220 2578 generic.go:356] "Generic (PLEG): container finished" podID="b77c3031-3b4e-4566-86ea-3b6f2db61c01" containerID="fd03089eaad52ab0000ef4f184bd7d68bbbe6e5069009c49abd33e0b52771582" exitCode=0 Jul 14 20:30:02.805688 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:02.805293 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" event={"ID":"b77c3031-3b4e-4566-86ea-3b6f2db61c01","Type":"ContainerDied","Data":"fd03089eaad52ab0000ef4f184bd7d68bbbe6e5069009c49abd33e0b52771582"} Jul 14 20:30:02.956408 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:02.956379 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n" Jul 14 20:30:02.994646 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:02.994616 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" Jul 14 20:30:03.812692 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:03.812655 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" Jul 14 20:30:03.813155 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:03.812651 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk" event={"ID":"0a2d1044-2625-4a5d-a72e-8ebab927a3bc","Type":"ContainerDied","Data":"dbf64a6196756d498d80c5cffdc8568f59440cf1bfbd561451e1a725aab4a2f6"} Jul 14 20:30:03.813155 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:03.812808 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbf64a6196756d498d80c5cffdc8568f59440cf1bfbd561451e1a725aab4a2f6" Jul 14 20:30:03.813939 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:03.813918 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n" Jul 14 20:30:03.814006 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:03.813958 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29734350-c9p4n" event={"ID":"5d662aee-601e-4faf-aa07-9629f8834ab9","Type":"ContainerDied","Data":"0f15daf27676ecf6d5c27ad3a000b11e1ab0599891be99ff42a39b29582269fc"} Jul 14 20:30:03.814006 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:03.814001 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f15daf27676ecf6d5c27ad3a000b11e1ab0599891be99ff42a39b29582269fc" Jul 14 20:30:03.972321 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:03.972285 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" Jul 14 20:30:04.822353 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:04.822315 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" Jul 14 20:30:04.822353 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:04.822340 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84" event={"ID":"b77c3031-3b4e-4566-86ea-3b6f2db61c01","Type":"ContainerDied","Data":"748d3a30d8f2ef9cacce5407c7c61f371f6d3d0ecedae763b72db8e0e99d2f80"} Jul 14 20:30:04.822814 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:04.822392 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="748d3a30d8f2ef9cacce5407c7c61f371f6d3d0ecedae763b72db8e0e99d2f80" Jul 14 20:30:35.108917 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.108879 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj"] Jul 14 20:30:35.109423 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109318 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d662aee-601e-4faf-aa07-9629f8834ab9" containerName="cleanup" Jul 14 20:30:35.109423 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109328 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d662aee-601e-4faf-aa07-9629f8834ab9" containerName="cleanup" Jul 14 20:30:35.109423 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109345 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77c3031-3b4e-4566-86ea-3b6f2db61c01" containerName="cleanup" Jul 14 20:30:35.109423 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109351 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77c3031-3b4e-4566-86ea-3b6f2db61c01" containerName="cleanup" Jul 14 20:30:35.109423 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109372 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2d1044-2625-4a5d-a72e-8ebab927a3bc" containerName="cleanup" Jul 14 20:30:35.109423 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109378 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2d1044-2625-4a5d-a72e-8ebab927a3bc" containerName="cleanup" Jul 14 20:30:35.109646 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109447 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d662aee-601e-4faf-aa07-9629f8834ab9" containerName="cleanup" Jul 14 20:30:35.109646 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109456 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b77c3031-3b4e-4566-86ea-3b6f2db61c01" containerName="cleanup" Jul 14 20:30:35.109646 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.109464 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2d1044-2625-4a5d-a72e-8ebab927a3bc" containerName="cleanup" Jul 14 20:30:35.112500 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.112483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.116294 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.116266 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"kube-root-ca.crt\"" Jul 14 20:30:35.116294 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.116278 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"default-dockercfg-pcpdr\"" Jul 14 20:30:35.116514 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.116307 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"openshift-service-ca.crt\"" Jul 14 20:30:35.116514 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.116426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fad346\"/\"rate-test-model-fad346-kserve-self-signed-certs\"" Jul 14 20:30:35.121884 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.121858 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj"] Jul 14 20:30:35.261565 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.261529 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-home\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.261565 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.261571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-tmp-dir\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.261889 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.261593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-dshm\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.261889 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.261673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-model-cache\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.261889 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.261732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92378939-1fa8-40c1-87c8-c49b5da970a5-tls-certs\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.261889 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.261762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82jzd\" (UniqueName: \"kubernetes.io/projected/92378939-1fa8-40c1-87c8-c49b5da970a5-kube-api-access-82jzd\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.362888 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.362754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-home\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.362888 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.362802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-tmp-dir\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.362888 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.362845 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-dshm\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.362888 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.362879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-model-cache\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.363254 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.362918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92378939-1fa8-40c1-87c8-c49b5da970a5-tls-certs\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.363254 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.362947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82jzd\" (UniqueName: \"kubernetes.io/projected/92378939-1fa8-40c1-87c8-c49b5da970a5-kube-api-access-82jzd\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.363254 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.363249 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-home\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.363399 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.363309 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-tmp-dir\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.363399 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.363378 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-model-cache\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.365527 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.365502 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-dshm\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.365642 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.365531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92378939-1fa8-40c1-87c8-c49b5da970a5-tls-certs\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.372116 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.372092 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82jzd\" (UniqueName: \"kubernetes.io/projected/92378939-1fa8-40c1-87c8-c49b5da970a5-kube-api-access-82jzd\") pod \"rate-test-model-fad346-kserve-d77d4c5-z8fzj\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.422533 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.422491 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:35.574393 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:35.574359 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj"] Jul 14 20:30:35.575164 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:30:35.575137 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92378939_1fa8_40c1_87c8_c49b5da970a5.slice/crio-ac16f72db4cabee23e8e8a804f1d8142850cef54f2ffc18f5bdd87cee3a7b77a WatchSource:0}: Error finding container ac16f72db4cabee23e8e8a804f1d8142850cef54f2ffc18f5bdd87cee3a7b77a: Status 404 returned error can't find the container with id ac16f72db4cabee23e8e8a804f1d8142850cef54f2ffc18f5bdd87cee3a7b77a Jul 14 20:30:36.003321 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:36.003226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" event={"ID":"92378939-1fa8-40c1-87c8-c49b5da970a5","Type":"ContainerStarted","Data":"0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6"} Jul 14 20:30:36.003321 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:36.003267 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" event={"ID":"92378939-1fa8-40c1-87c8-c49b5da970a5","Type":"ContainerStarted","Data":"ac16f72db4cabee23e8e8a804f1d8142850cef54f2ffc18f5bdd87cee3a7b77a"} Jul 14 20:30:36.024143 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:36.024083 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" podStartSLOduration=1.024067855 podStartE2EDuration="1.024067855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:30:36.021754559 +0000 UTC m=+2677.669318125" watchObservedRunningTime="2026-07-14 20:30:36.024067855 +0000 UTC m=+2677.671631421" Jul 14 20:30:45.422803 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:45.422754 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:45.423236 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:45.422822 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:45.436472 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:45.436439 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:46.071953 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:46.071920 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:30:59.048891 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:59.048791 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:30:59.055243 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:59.055211 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:30:59.060997 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:59.060972 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:30:59.066965 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:30:59.066943 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kkjx6_fb74204e-abe8-45da-8bbd-97eb68965a34/ovn-acl-logging/0.log" Jul 14 20:32:13.167223 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:13.167188 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj"] Jul 14 20:32:21.720006 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:21.719916 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:21.723672 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:21.723652 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:21.726640 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:21.726612 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v9vdk\"" Jul 14 20:32:21.729530 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:21.729501 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:21.777036 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:21.776999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2pr\" (UniqueName: \"kubernetes.io/projected/22577e33-ee03-4ae5-9a9a-729254496ceb-kube-api-access-9r2pr\") pod \"test-curl-7502-140498342228016\" (UID: \"22577e33-ee03-4ae5-9a9a-729254496ceb\") " pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:21.877676 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:21.877634 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2pr\" (UniqueName: \"kubernetes.io/projected/22577e33-ee03-4ae5-9a9a-729254496ceb-kube-api-access-9r2pr\") pod \"test-curl-7502-140498342228016\" (UID: \"22577e33-ee03-4ae5-9a9a-729254496ceb\") " pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:21.887862 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:21.887794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2pr\" (UniqueName: \"kubernetes.io/projected/22577e33-ee03-4ae5-9a9a-729254496ceb-kube-api-access-9r2pr\") pod \"test-curl-7502-140498342228016\" (UID: \"22577e33-ee03-4ae5-9a9a-729254496ceb\") " pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:22.034141 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:22.034043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:22.175463 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:22.175429 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:22.177199 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:32:22.177170 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22577e33_ee03_4ae5_9a9a_729254496ceb.slice/crio-a7b6ef219018cc58513452928dc3c3365222d5ed6216db4799139f0d0e38d656 WatchSource:0}: Error finding container a7b6ef219018cc58513452928dc3c3365222d5ed6216db4799139f0d0e38d656: Status 404 returned error can't find the container with id a7b6ef219018cc58513452928dc3c3365222d5ed6216db4799139f0d0e38d656 Jul 14 20:32:22.178648 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:22.178627 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 20:32:22.639470 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:22.639436 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342228016" event={"ID":"22577e33-ee03-4ae5-9a9a-729254496ceb","Type":"ContainerStarted","Data":"a7b6ef219018cc58513452928dc3c3365222d5ed6216db4799139f0d0e38d656"} Jul 14 20:32:23.646897 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:23.646859 2578 generic.go:356] "Generic (PLEG): container finished" podID="22577e33-ee03-4ae5-9a9a-729254496ceb" containerID="4b3222fc809f6869db3ff29a57c2b85e6f6d078f92869e5a9ec2ed7a4115f238" exitCode=0 Jul 14 20:32:23.647296 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:23.646935 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342228016" event={"ID":"22577e33-ee03-4ae5-9a9a-729254496ceb","Type":"ContainerDied","Data":"4b3222fc809f6869db3ff29a57c2b85e6f6d078f92869e5a9ec2ed7a4115f238"} Jul 14 20:32:24.795544 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:24.795512 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:24.830717 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:24.830683 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7502-140498342228016_22577e33-ee03-4ae5-9a9a-729254496ceb/test-curl-7502-140498342228016/0.log" Jul 14 20:32:24.860970 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:24.860934 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:24.867014 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:24.866983 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:24.898654 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:24.898626 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r2pr\" (UniqueName: \"kubernetes.io/projected/22577e33-ee03-4ae5-9a9a-729254496ceb-kube-api-access-9r2pr\") pod \"22577e33-ee03-4ae5-9a9a-729254496ceb\" (UID: \"22577e33-ee03-4ae5-9a9a-729254496ceb\") " Jul 14 20:32:24.900871 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:24.900842 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22577e33-ee03-4ae5-9a9a-729254496ceb-kube-api-access-9r2pr" (OuterVolumeSpecName: "kube-api-access-9r2pr") pod "22577e33-ee03-4ae5-9a9a-729254496ceb" (UID: "22577e33-ee03-4ae5-9a9a-729254496ceb"). InnerVolumeSpecName "kube-api-access-9r2pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:24.999613 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:24.999570 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2pr\" (UniqueName: \"kubernetes.io/projected/22577e33-ee03-4ae5-9a9a-729254496ceb-kube-api-access-9r2pr\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:25.019743 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.019702 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:25.020165 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.020151 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22577e33-ee03-4ae5-9a9a-729254496ceb" containerName="test-curl-7502-140498342228016" Jul 14 20:32:25.020217 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.020167 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="22577e33-ee03-4ae5-9a9a-729254496ceb" containerName="test-curl-7502-140498342228016" Jul 14 20:32:25.020249 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.020235 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="22577e33-ee03-4ae5-9a9a-729254496ceb" containerName="test-curl-7502-140498342228016" Jul 14 20:32:25.023536 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.023505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:25.028995 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.028967 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:25.100699 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.100656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgct8\" (UniqueName: \"kubernetes.io/projected/b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d-kube-api-access-zgct8\") pod \"test-curl-7502-140498342228016\" (UID: \"b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d\") " pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:25.202007 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.201905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgct8\" (UniqueName: \"kubernetes.io/projected/b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d-kube-api-access-zgct8\") pod \"test-curl-7502-140498342228016\" (UID: \"b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d\") " pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:25.211930 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.211891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgct8\" (UniqueName: \"kubernetes.io/projected/b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d-kube-api-access-zgct8\") pod \"test-curl-7502-140498342228016\" (UID: \"b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d\") " pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:25.335222 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.335165 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:25.493423 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.493395 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:25.495247 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:32:25.495220 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0f273e4_9cc4_4014_b9ca_8c5d8a4ff35d.slice/crio-81d57056976b6cc706efe57c2601bd6356ca0a08f0cc234f548bf6a8b3c62b4d WatchSource:0}: Error finding container 81d57056976b6cc706efe57c2601bd6356ca0a08f0cc234f548bf6a8b3c62b4d: Status 404 returned error can't find the container with id 81d57056976b6cc706efe57c2601bd6356ca0a08f0cc234f548bf6a8b3c62b4d Jul 14 20:32:25.659181 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.659137 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342228016" event={"ID":"b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d","Type":"ContainerStarted","Data":"81d57056976b6cc706efe57c2601bd6356ca0a08f0cc234f548bf6a8b3c62b4d"} Jul 14 20:32:25.660465 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.660444 2578 scope.go:117] "RemoveContainer" containerID="4b3222fc809f6869db3ff29a57c2b85e6f6d078f92869e5a9ec2ed7a4115f238" Jul 14 20:32:25.660465 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:25.660470 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:26.667635 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:26.667597 2578 generic.go:356] "Generic (PLEG): container finished" podID="b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" containerID="fb7d13f5285d8bab8ac91643c4a66f7df0fcb0db3752aa3a842fca4bd1bdc674" exitCode=0 Jul 14 20:32:26.668106 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:26.667695 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7502-140498342228016" event={"ID":"b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d","Type":"ContainerDied","Data":"fb7d13f5285d8bab8ac91643c4a66f7df0fcb0db3752aa3a842fca4bd1bdc674"} Jul 14 20:32:26.884181 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:26.884148 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22577e33-ee03-4ae5-9a9a-729254496ceb" path="/var/lib/kubelet/pods/22577e33-ee03-4ae5-9a9a-729254496ceb/volumes" Jul 14 20:32:27.822249 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:27.822221 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:27.844851 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:27.844805 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7502-140498342228016_b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d/test-curl-7502-140498342228016/0.log" Jul 14 20:32:27.881290 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:27.880180 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:27.886441 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:27.886398 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7502-140498342228016"] Jul 14 20:32:27.927137 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:27.927041 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgct8\" (UniqueName: \"kubernetes.io/projected/b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d-kube-api-access-zgct8\") pod \"b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d\" (UID: \"b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d\") " Jul 14 20:32:27.929259 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:27.929223 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d-kube-api-access-zgct8" (OuterVolumeSpecName: "kube-api-access-zgct8") pod "b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" (UID: "b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d"). InnerVolumeSpecName "kube-api-access-zgct8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:28.028733 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.028692 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgct8\" (UniqueName: \"kubernetes.io/projected/b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d-kube-api-access-zgct8\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:28.188702 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.188604 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" podUID="92378939-1fa8-40c1-87c8-c49b5da970a5" containerName="main" containerID="cri-o://0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6" gracePeriod=45 Jul 14 20:32:28.458994 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.458968 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:32:28.533203 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533160 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-tmp-dir\") pod \"92378939-1fa8-40c1-87c8-c49b5da970a5\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " Jul 14 20:32:28.533425 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533219 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92378939-1fa8-40c1-87c8-c49b5da970a5-tls-certs\") pod \"92378939-1fa8-40c1-87c8-c49b5da970a5\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " Jul 14 20:32:28.533425 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533248 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82jzd\" (UniqueName: \"kubernetes.io/projected/92378939-1fa8-40c1-87c8-c49b5da970a5-kube-api-access-82jzd\") pod \"92378939-1fa8-40c1-87c8-c49b5da970a5\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " Jul 14 20:32:28.533425 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533322 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-dshm\") pod \"92378939-1fa8-40c1-87c8-c49b5da970a5\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " Jul 14 20:32:28.533425 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533351 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-home\") pod \"92378939-1fa8-40c1-87c8-c49b5da970a5\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " Jul 14 20:32:28.533425 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533407 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-model-cache\") pod \"92378939-1fa8-40c1-87c8-c49b5da970a5\" (UID: \"92378939-1fa8-40c1-87c8-c49b5da970a5\") " Jul 14 20:32:28.533712 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533521 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92378939-1fa8-40c1-87c8-c49b5da970a5" (UID: "92378939-1fa8-40c1-87c8-c49b5da970a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:28.533766 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533726 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "92378939-1fa8-40c1-87c8-c49b5da970a5" (UID: "92378939-1fa8-40c1-87c8-c49b5da970a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:28.533813 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533795 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-model-cache\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:28.533908 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533811 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-tmp-dir\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:28.533967 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.533939 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-home" (OuterVolumeSpecName: "home") pod "92378939-1fa8-40c1-87c8-c49b5da970a5" (UID: "92378939-1fa8-40c1-87c8-c49b5da970a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:28.535771 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.535730 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92378939-1fa8-40c1-87c8-c49b5da970a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92378939-1fa8-40c1-87c8-c49b5da970a5" (UID: "92378939-1fa8-40c1-87c8-c49b5da970a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 20:32:28.535922 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.535870 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92378939-1fa8-40c1-87c8-c49b5da970a5-kube-api-access-82jzd" (OuterVolumeSpecName: "kube-api-access-82jzd") pod "92378939-1fa8-40c1-87c8-c49b5da970a5" (UID: "92378939-1fa8-40c1-87c8-c49b5da970a5"). InnerVolumeSpecName "kube-api-access-82jzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 20:32:28.535922 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.535880 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-dshm" (OuterVolumeSpecName: "dshm") pod "92378939-1fa8-40c1-87c8-c49b5da970a5" (UID: "92378939-1fa8-40c1-87c8-c49b5da970a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 20:32:28.634692 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.634643 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92378939-1fa8-40c1-87c8-c49b5da970a5-tls-certs\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:28.634692 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.634683 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82jzd\" (UniqueName: \"kubernetes.io/projected/92378939-1fa8-40c1-87c8-c49b5da970a5-kube-api-access-82jzd\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:28.634692 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.634696 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-dshm\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:28.634998 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.634708 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92378939-1fa8-40c1-87c8-c49b5da970a5-home\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:28.680812 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.680780 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:28.681011 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.680815 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d57056976b6cc706efe57c2601bd6356ca0a08f0cc234f548bf6a8b3c62b4d" Jul 14 20:32:28.682247 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.682220 2578 generic.go:356] "Generic (PLEG): container finished" podID="92378939-1fa8-40c1-87c8-c49b5da970a5" containerID="0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6" exitCode=0 Jul 14 20:32:28.682399 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.682266 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" Jul 14 20:32:28.682399 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.682282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" event={"ID":"92378939-1fa8-40c1-87c8-c49b5da970a5","Type":"ContainerDied","Data":"0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6"} Jul 14 20:32:28.682399 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.682321 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj" event={"ID":"92378939-1fa8-40c1-87c8-c49b5da970a5","Type":"ContainerDied","Data":"ac16f72db4cabee23e8e8a804f1d8142850cef54f2ffc18f5bdd87cee3a7b77a"} Jul 14 20:32:28.682399 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.682341 2578 scope.go:117] "RemoveContainer" containerID="0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6" Jul 14 20:32:28.692575 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.692551 2578 scope.go:117] "RemoveContainer" containerID="0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6" Jul 14 20:32:28.692908 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:32:28.692890 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6\": container with ID starting with 0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6 not found: ID does not exist" containerID="0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6" Jul 14 20:32:28.692991 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.692916 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6"} err="failed to get container status \"0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6\": rpc error: code = NotFound desc = could not find container \"0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6\": container with ID starting with 0e1ac2a825f5070ee929cb145aea2570728e0c109bb3b0b57803ded789a143e6 not found: ID does not exist" Jul 14 20:32:28.708205 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.708170 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj"] Jul 14 20:32:28.713405 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:32:28.713363 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7502-140498342228016\" is forbidden: User \"system:node:ip-10-0-133-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-243.ec2.internal' and this object" podUID="b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:28.714599 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.714576 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fad346/rate-test-model-fad346-kserve-d77d4c5-z8fzj"] Jul 14 20:32:28.715407 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:32:28.715384 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7502-140498342228016\" is forbidden: User \"system:node:ip-10-0-133-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-243.ec2.internal' and this object" podUID="b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:28.883067 ip-10-0-133-243 kubenswrapper[2578]: E0714 20:32:28.883023 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7502-140498342228016\" is forbidden: User \"system:node:ip-10-0-133-243.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-243.ec2.internal' and this object" podUID="b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" pod="opendatahub/test-curl-7502-140498342228016" Jul 14 20:32:28.884772 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.884742 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92378939-1fa8-40c1-87c8-c49b5da970a5" path="/var/lib/kubelet/pods/92378939-1fa8-40c1-87c8-c49b5da970a5/volumes" Jul 14 20:32:28.885198 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:28.885176 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" path="/var/lib/kubelet/pods/b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d/volumes" Jul 14 20:32:33.803528 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.803479 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb"] Jul 14 20:32:33.804138 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.804115 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92378939-1fa8-40c1-87c8-c49b5da970a5" containerName="main" Jul 14 20:32:33.804222 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.804140 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92378939-1fa8-40c1-87c8-c49b5da970a5" containerName="main" Jul 14 20:32:33.804222 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.804161 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" containerName="test-curl-7502-140498342228016" Jul 14 20:32:33.804222 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.804170 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" containerName="test-curl-7502-140498342228016" Jul 14 20:32:33.804374 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.804300 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92378939-1fa8-40c1-87c8-c49b5da970a5" containerName="main" Jul 14 20:32:33.804374 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.804318 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f273e4-9cc4-4014-b9ca-8c5d8a4ff35d" containerName="test-curl-7502-140498342228016" Jul 14 20:32:33.808897 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.808870 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" Jul 14 20:32:33.811368 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.811293 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:32:33.811368 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.811366 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:32:33.816544 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.816497 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb"] Jul 14 20:32:33.883275 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.883225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4000f00a-5538-4ae7-bae1-205d9460784f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb\" (UID: \"4000f00a-5538-4ae7-bae1-205d9460784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" Jul 14 20:32:33.984484 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.984443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4000f00a-5538-4ae7-bae1-205d9460784f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb\" (UID: \"4000f00a-5538-4ae7-bae1-205d9460784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" Jul 14 20:32:33.985173 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:33.985153 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4000f00a-5538-4ae7-bae1-205d9460784f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb\" (UID: \"4000f00a-5538-4ae7-bae1-205d9460784f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" Jul 14 20:32:34.122435 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:34.122398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" Jul 14 20:32:34.478798 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:34.478762 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb"] Jul 14 20:32:34.480454 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:32:34.480426 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4000f00a_5538_4ae7_bae1_205d9460784f.slice/crio-ef705d4733fc751c2f68dcaaf3ee68baab5caf62c7db42543b16f8bc740f2a52 WatchSource:0}: Error finding container ef705d4733fc751c2f68dcaaf3ee68baab5caf62c7db42543b16f8bc740f2a52: Status 404 returned error can't find the container with id ef705d4733fc751c2f68dcaaf3ee68baab5caf62c7db42543b16f8bc740f2a52 Jul 14 20:32:34.721455 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:34.721421 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" event={"ID":"4000f00a-5538-4ae7-bae1-205d9460784f","Type":"ContainerStarted","Data":"ef705d4733fc751c2f68dcaaf3ee68baab5caf62c7db42543b16f8bc740f2a52"} Jul 14 20:32:35.730186 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:35.730144 2578 generic.go:356] "Generic (PLEG): container finished" podID="4000f00a-5538-4ae7-bae1-205d9460784f" containerID="8f9b8d83913947c88e8f69df1027b001431fccb8bd3ae83b5394b299e4ad1fd6" exitCode=0 Jul 14 20:32:35.730664 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:35.730215 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" event={"ID":"4000f00a-5538-4ae7-bae1-205d9460784f","Type":"ContainerDied","Data":"8f9b8d83913947c88e8f69df1027b001431fccb8bd3ae83b5394b299e4ad1fd6"} Jul 14 20:32:36.887253 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:36.887224 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" Jul 14 20:32:37.015552 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:37.015454 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4000f00a-5538-4ae7-bae1-205d9460784f-maas-api-service-ca\") pod \"4000f00a-5538-4ae7-bae1-205d9460784f\" (UID: \"4000f00a-5538-4ae7-bae1-205d9460784f\") " Jul 14 20:32:37.015914 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:37.015892 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4000f00a-5538-4ae7-bae1-205d9460784f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4000f00a-5538-4ae7-bae1-205d9460784f" (UID: "4000f00a-5538-4ae7-bae1-205d9460784f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:32:37.116429 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:37.116390 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4000f00a-5538-4ae7-bae1-205d9460784f-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:32:37.743037 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:37.743004 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" Jul 14 20:32:37.743227 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:37.743011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb" event={"ID":"4000f00a-5538-4ae7-bae1-205d9460784f","Type":"ContainerDied","Data":"ef705d4733fc751c2f68dcaaf3ee68baab5caf62c7db42543b16f8bc740f2a52"} Jul 14 20:32:37.743227 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:37.743131 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef705d4733fc751c2f68dcaaf3ee68baab5caf62c7db42543b16f8bc740f2a52" Jul 14 20:32:44.322137 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:44.322096 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84"] Jul 14 20:32:44.325939 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:44.325909 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fad346-29734350-zvd84"] Jul 14 20:32:44.883852 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:44.883801 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b77c3031-3b4e-4566-86ea-3b6f2db61c01" path="/var/lib/kubelet/pods/b77c3031-3b4e-4566-86ea-3b6f2db61c01/volumes" Jul 14 20:32:59.990070 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:59.990033 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll"] Jul 14 20:32:59.990638 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:59.990616 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4000f00a-5538-4ae7-bae1-205d9460784f" containerName="revoke-keys" Jul 14 20:32:59.990729 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:59.990641 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4000f00a-5538-4ae7-bae1-205d9460784f" containerName="revoke-keys" Jul 14 20:32:59.990791 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:59.990780 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4000f00a-5538-4ae7-bae1-205d9460784f" containerName="revoke-keys" Jul 14 20:32:59.995817 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:59.995791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" Jul 14 20:32:59.998776 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:59.998744 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 20:32:59.998954 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:32:59.998907 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9ztwd\"" Jul 14 20:33:00.001203 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.001175 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll"] Jul 14 20:33:00.148034 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.147990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcbd19e-dcf8-482e-9494-1c688a930ab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll\" (UID: \"abcbd19e-dcf8-482e-9494-1c688a930ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" Jul 14 20:33:00.248790 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.248703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcbd19e-dcf8-482e-9494-1c688a930ab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll\" (UID: \"abcbd19e-dcf8-482e-9494-1c688a930ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" Jul 14 20:33:00.249320 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.249300 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcbd19e-dcf8-482e-9494-1c688a930ab3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll\" (UID: \"abcbd19e-dcf8-482e-9494-1c688a930ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" Jul 14 20:33:00.309625 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.309586 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" Jul 14 20:33:00.456937 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:33:00.456898 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcbd19e_dcf8_482e_9494_1c688a930ab3.slice/crio-f45ff1f06bbf4fbe212c4690e83b3ef76a4170bc48b2c265ecfd5b83901419c3 WatchSource:0}: Error finding container f45ff1f06bbf4fbe212c4690e83b3ef76a4170bc48b2c265ecfd5b83901419c3: Status 404 returned error can't find the container with id f45ff1f06bbf4fbe212c4690e83b3ef76a4170bc48b2c265ecfd5b83901419c3 Jul 14 20:33:00.457090 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.457061 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll"] Jul 14 20:33:00.883722 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.883687 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" event={"ID":"abcbd19e-dcf8-482e-9494-1c688a930ab3","Type":"ContainerStarted","Data":"deb32756d53f60f7117ab3044ef882328988ece616f0e8ccb1aa523a65f4278d"} Jul 14 20:33:00.883933 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.883744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" event={"ID":"abcbd19e-dcf8-482e-9494-1c688a930ab3","Type":"ContainerStarted","Data":"f45ff1f06bbf4fbe212c4690e83b3ef76a4170bc48b2c265ecfd5b83901419c3"} Jul 14 20:33:00.897552 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:00.897442 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" podStartSLOduration=1.897424824 podStartE2EDuration="1.897424824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 20:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:33:00.895621401 +0000 UTC m=+2822.543184983" watchObservedRunningTime="2026-07-14 20:33:00.897424824 +0000 UTC m=+2822.544988389" Jul 14 20:33:01.886589 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:01.886556 2578 generic.go:356] "Generic (PLEG): container finished" podID="abcbd19e-dcf8-482e-9494-1c688a930ab3" containerID="deb32756d53f60f7117ab3044ef882328988ece616f0e8ccb1aa523a65f4278d" exitCode=0 Jul 14 20:33:01.887017 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:01.886627 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" event={"ID":"abcbd19e-dcf8-482e-9494-1c688a930ab3","Type":"ContainerDied","Data":"deb32756d53f60f7117ab3044ef882328988ece616f0e8ccb1aa523a65f4278d"} Jul 14 20:33:03.041242 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:03.041200 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" Jul 14 20:33:03.072023 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:03.071990 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcbd19e-dcf8-482e-9494-1c688a930ab3-maas-api-service-ca\") pod \"abcbd19e-dcf8-482e-9494-1c688a930ab3\" (UID: \"abcbd19e-dcf8-482e-9494-1c688a930ab3\") " Jul 14 20:33:03.072350 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:03.072322 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcbd19e-dcf8-482e-9494-1c688a930ab3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abcbd19e-dcf8-482e-9494-1c688a930ab3" (UID: "abcbd19e-dcf8-482e-9494-1c688a930ab3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 20:33:03.173484 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:03.173389 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abcbd19e-dcf8-482e-9494-1c688a930ab3-maas-api-service-ca\") on node \"ip-10-0-133-243.ec2.internal\" DevicePath \"\"" Jul 14 20:33:03.899586 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:03.899551 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" Jul 14 20:33:03.899775 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:03.899548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll" event={"ID":"abcbd19e-dcf8-482e-9494-1c688a930ab3","Type":"ContainerDied","Data":"f45ff1f06bbf4fbe212c4690e83b3ef76a4170bc48b2c265ecfd5b83901419c3"} Jul 14 20:33:03.899775 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:03.899675 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f45ff1f06bbf4fbe212c4690e83b3ef76a4170bc48b2c265ecfd5b83901419c3" Jul 14 20:33:10.482180 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:10.482146 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk"] Jul 14 20:33:10.485104 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:10.485076 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a99cbd-29734350-ctlpk"] Jul 14 20:33:10.884666 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:10.884633 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2d1044-2625-4a5d-a72e-8ebab927a3bc" path="/var/lib/kubelet/pods/0a2d1044-2625-4a5d-a72e-8ebab927a3bc/volumes" Jul 14 20:33:31.405739 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:31.405705 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lc6qm_99de2193-68ea-4c3f-bf6a-efa88689502a/manager/0.log" Jul 14 20:33:31.531704 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:31.531668 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fh5pg_f1674e59-0ac4-4f18-8047-4de1c2793908/manager/0.log" Jul 14 20:33:31.772035 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:31.771950 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bcd84_d8a97c62-1a40-4ab4-b6be-25580d851593/server/0.log" Jul 14 20:33:31.898188 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:31.898150 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rgc5q_0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4/manager/0.log" Jul 14 20:33:32.149515 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:32.149485 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-mkd87_fd42ad44-f326-448d-8795-71faf345277d/copy-manifests/0.log" Jul 14 20:33:32.157411 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:32.157379 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-mkd87_fd42ad44-f326-448d-8795-71faf345277d/manager/0.log" Jul 14 20:33:32.917934 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:32.917893 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734335-879vp_ddefaeae-898d-430d-aec8-2ff4df478ab1/cleanup/0.log" Jul 14 20:33:33.039600 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:33.039568 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29734350-c9p4n_5d662aee-601e-4faf-aa07-9629f8834ab9/cleanup/0.log" Jul 14 20:33:33.160919 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:33.160893 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-ef48f8e1-mgqjg_7448882c-f190-40f6-bc80-9010e2d9a768/revoke-keys/1.log" Jul 14 20:33:33.282916 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:33.282794 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-f61d89f4-ht5j7_32eb0459-918c-4a9d-87b9-16cc92d00d6f/revoke-keys/0.log" Jul 14 20:33:33.394147 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:33.394107 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-285387-42pqq_6ed018c8-e271-48c8-be16-28a5f8678b6a/revoke-keys/0.log" Jul 14 20:33:33.507612 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:33.507582 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-d3ee05-bhp75_b4f96987-ef73-4641-bef9-7a98f8a5858c/revoke-keys/0.log" Jul 14 20:33:33.625059 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:33.625027 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-e772d17e-mdr2n_349ada76-32c3-4383-86cd-7f7bd7614dcf/revoke-keys/0.log" Jul 14 20:33:33.743673 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:33.743644 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-1c67fa-m2bgc_f38bf502-a8ed-4eb4-ae77-09d62fcd7f04/revoke-keys/0.log" Jul 14 20:33:33.858278 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:33.858245 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-2e7e45-dckw6_5c112c97-6433-4578-8d68-b4863cdc2b50/revoke-keys/0.log" Jul 14 20:33:34.100108 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:34.100072 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-4d0ddec0-dktlq_e64144d3-a7f2-4800-8e03-74354b3cdbd2/revoke-keys/0.log" Jul 14 20:33:34.214459 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:34.214429 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-ad8efdd2-nh7ct_4137873f-2cc5-42a3-9907-ab980589017c/revoke-keys/0.log" Jul 14 20:33:34.327066 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:34.327036 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-fad346-dc9hb_4000f00a-5538-4ae7-bae1-205d9460784f/revoke-keys/0.log" Jul 14 20:33:34.438088 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:34.438008 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-a99cbd-fh5ll_abcbd19e-dcf8-482e-9494-1c688a930ab3/revoke-keys/0.log" Jul 14 20:33:35.984777 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:35.984748 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ndrxf_a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f/manager/0.log" Jul 14 20:33:36.096743 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:36.096708 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5qng7_13b6802a-e0f2-4e7a-a79b-5994b4f5ca85/manager/0.log" Jul 14 20:33:36.209994 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:36.209959 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pkl49_a56f7b6f-6794-457e-aae9-9422e3e8c243/kuadrant-console-plugin/0.log" Jul 14 20:33:36.330597 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:36.330561 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dnc6t_29843439-dd60-4557-a0b9-6db34a36a6a9/registry-server/0.log" Jul 14 20:33:36.464071 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:36.464040 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-j2z2l_85fc64a4-8926-4e1b-a493-538afe904263/manager/0.log" Jul 14 20:33:37.109398 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:37.109363 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf494s6n_9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c/istio-proxy/0.log" Jul 14 20:33:37.347044 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:37.347014 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5df66fb8c-jh5t7_2e58882e-0431-4140-9998-3303528d1457/kube-auth-proxy/0.log" Jul 14 20:33:37.605661 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:37.605623 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pthzr_a42a78ef-b193-4fc1-8196-2b1979b4a50f/istio-proxy/1.log" Jul 14 20:33:37.732177 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:37.732144 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-flfq6_a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4/payload-pre-processing/0.log" Jul 14 20:33:37.854063 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:37.854030 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-vjgzw_456b87f2-0077-4159-8049-4eae24c13fd9/payload-processing/0.log" Jul 14 20:33:37.964162 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:37.964078 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b7cc8dd66-pmcdg_acafe8a7-3da1-4bac-b4a9-81aaa72fc21f/router/0.log" Jul 14 20:33:38.569938 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:38.569906 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-782qh_9a8a8afd-9c65-46b4-931c-94e0c774b2d7/main/0.log" Jul 14 20:33:38.817146 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:38.817110 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-58mq2_21118a9c-ef38-4929-ae38-bf1e14e0be71/main/0.log" Jul 14 20:33:46.515769 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:46.515726 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qzk58_060c531f-952a-432e-a4a3-c08cd392bdcd/global-pull-secret-syncer/0.log" Jul 14 20:33:46.608992 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:46.608961 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-65rs9_d17644b4-19e1-4aa9-b8d3-10e6062c5ffd/konnectivity-agent/0.log" Jul 14 20:33:46.693617 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:46.693578 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-243.ec2.internal_f9010125a2cabcaaf81321ade9e08441/haproxy/0.log" Jul 14 20:33:52.045986 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:52.045946 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ndrxf_a1a6291d-fe14-4f8f-838d-9dc5c7b19f7f/manager/0.log" Jul 14 20:33:52.074106 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:52.074078 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5qng7_13b6802a-e0f2-4e7a-a79b-5994b4f5ca85/manager/0.log" Jul 14 20:33:52.102242 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:52.102217 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pkl49_a56f7b6f-6794-457e-aae9-9422e3e8c243/kuadrant-console-plugin/0.log" Jul 14 20:33:52.166030 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:52.165998 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dnc6t_29843439-dd60-4557-a0b9-6db34a36a6a9/registry-server/0.log" Jul 14 20:33:52.540095 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:52.540060 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-j2z2l_85fc64a4-8926-4e1b-a493-538afe904263/manager/0.log" Jul 14 20:33:54.533489 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:54.533455 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-f4mvb_efee90be-3aa3-43a7-9a09-9a537528cff0/cluster-monitoring-operator/0.log" Jul 14 20:33:54.887615 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:54.887584 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqq6d_595ebd05-e643-4645-9025-b3bddb2e02db/node-exporter/0.log" Jul 14 20:33:54.932518 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:54.932475 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqq6d_595ebd05-e643-4645-9025-b3bddb2e02db/kube-rbac-proxy/0.log" Jul 14 20:33:54.979644 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:54.979615 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqq6d_595ebd05-e643-4645-9025-b3bddb2e02db/init-textfile/0.log" Jul 14 20:33:56.449658 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.449623 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6"] Jul 14 20:33:56.450134 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.450104 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abcbd19e-dcf8-482e-9494-1c688a930ab3" containerName="revoke-keys" Jul 14 20:33:56.450134 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.450122 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="abcbd19e-dcf8-482e-9494-1c688a930ab3" containerName="revoke-keys" Jul 14 20:33:56.450265 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.450250 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="abcbd19e-dcf8-482e-9494-1c688a930ab3" containerName="revoke-keys" Jul 14 20:33:56.453649 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.453629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.456390 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.456370 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v52j\"/\"kube-root-ca.crt\"" Jul 14 20:33:56.456535 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.456402 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7v52j\"/\"openshift-service-ca.crt\"" Jul 14 20:33:56.456535 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.456438 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7v52j\"/\"default-dockercfg-tbpgw\"" Jul 14 20:33:56.464756 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.464734 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6"] Jul 14 20:33:56.466953 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.466932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-proc\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.467054 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.466962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxt4q\" (UniqueName: \"kubernetes.io/projected/22504bcc-6794-4226-83a5-3ea8516506ce-kube-api-access-hxt4q\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.467054 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.466985 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-podres\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.467054 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.467034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-sys\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.467213 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.467106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-lib-modules\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568336 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-lib-modules\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568549 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-proc\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568549 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxt4q\" (UniqueName: \"kubernetes.io/projected/22504bcc-6794-4226-83a5-3ea8516506ce-kube-api-access-hxt4q\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568549 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-podres\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568549 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568474 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-lib-modules\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568549 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-proc\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568549 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-sys\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568791 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568591 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-podres\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.568791 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.568625 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22504bcc-6794-4226-83a5-3ea8516506ce-sys\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.578239 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.578213 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxt4q\" (UniqueName: \"kubernetes.io/projected/22504bcc-6794-4226-83a5-3ea8516506ce-kube-api-access-hxt4q\") pod \"perf-node-gather-daemonset-t4vd6\" (UID: \"22504bcc-6794-4226-83a5-3ea8516506ce\") " pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.723038 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.722952 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-nw56f_07fb4d56-0119-45b4-b2e7-db15a937702c/networking-console-plugin/0.log" Jul 14 20:33:56.764959 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.764927 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:56.905399 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:56.905369 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6"] Jul 14 20:33:56.906841 ip-10-0-133-243 kubenswrapper[2578]: W0714 20:33:56.906805 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod22504bcc_6794_4226_83a5_3ea8516506ce.slice/crio-c8dc70c6e44cdbb676f72452a32cb6d46b4483bada4e31abbbb93b54ad745087 WatchSource:0}: Error finding container c8dc70c6e44cdbb676f72452a32cb6d46b4483bada4e31abbbb93b54ad745087: Status 404 returned error can't find the container with id c8dc70c6e44cdbb676f72452a32cb6d46b4483bada4e31abbbb93b54ad745087 Jul 14 20:33:57.224101 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:57.224065 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" event={"ID":"22504bcc-6794-4226-83a5-3ea8516506ce","Type":"ContainerStarted","Data":"c0122e648a91fc6699e608e6611b5109e6063f2238029011464d3c55d9d02ae7"} Jul 14 20:33:57.224287 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:57.224122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" event={"ID":"22504bcc-6794-4226-83a5-3ea8516506ce","Type":"ContainerStarted","Data":"c8dc70c6e44cdbb676f72452a32cb6d46b4483bada4e31abbbb93b54ad745087"} Jul 14 20:33:57.224287 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:57.224196 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:33:57.242317 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:57.242269 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" podStartSLOduration=1.24225298 podStartE2EDuration="1.24225298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 20:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 20:33:57.239719677 +0000 UTC m=+2878.887283243" watchObservedRunningTime="2026-07-14 20:33:57.24225298 +0000 UTC m=+2878.889816545" Jul 14 20:33:57.283878 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:57.283849 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/1.log" Jul 14 20:33:57.294246 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:57.294170 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-h7mpd_371cf92f-1fd2-4277-a2b0-bd5fb0bb2f97/console-operator/2.log" Jul 14 20:33:57.803026 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:57.802991 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-t88pd_1e79965c-8b4b-4093-b6d7-e9789c8dbb8e/download-server/0.log" Jul 14 20:33:58.369489 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:58.369459 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-j62t4_40c179c1-a98f-437a-874a-ebb88a6b8200/volume-data-source-validator/0.log" Jul 14 20:33:59.450549 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:59.450517 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lb7wt_94fdd086-ee53-49c0-af48-d2602acad011/dns/0.log" Jul 14 20:33:59.474941 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:59.474913 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lb7wt_94fdd086-ee53-49c0-af48-d2602acad011/kube-rbac-proxy/0.log" Jul 14 20:33:59.607675 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:33:59.607633 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xthjn_18ce272c-d523-4253-bf80-92b16152b6e5/dns-node-resolver/0.log" Jul 14 20:34:00.239293 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:00.239266 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w2p7r_a52800f2-1350-49d4-ad60-df31b5d00640/node-ca/0.log" Jul 14 20:34:01.206316 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:01.206270 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf494s6n_9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c/istio-proxy/0.log" Jul 14 20:34:01.378667 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:01.378635 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5df66fb8c-jh5t7_2e58882e-0431-4140-9998-3303528d1457/kube-auth-proxy/0.log" Jul 14 20:34:01.437868 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:01.437797 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pthzr_a42a78ef-b193-4fc1-8196-2b1979b4a50f/istio-proxy/0.log" Jul 14 20:34:01.498641 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:01.498556 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pthzr_a42a78ef-b193-4fc1-8196-2b1979b4a50f/istio-proxy/1.log" Jul 14 20:34:01.538338 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:01.538310 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-flfq6_a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4/payload-pre-processing/0.log" Jul 14 20:34:01.577518 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:01.577485 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-vjgzw_456b87f2-0077-4159-8049-4eae24c13fd9/payload-processing/0.log" Jul 14 20:34:01.602373 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:01.602334 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b7cc8dd66-pmcdg_acafe8a7-3da1-4bac-b4a9-81aaa72fc21f/router/0.log" Jul 14 20:34:02.345359 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:02.345327 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lhn87_3dc21d0b-1485-4386-8f3b-ebdef6134092/serve-healthcheck-canary/0.log" Jul 14 20:34:02.834004 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:02.833963 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-6wwg6_85b2851b-b372-4935-9b8b-379379b4ae92/insights-operator/0.log" Jul 14 20:34:02.835638 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:02.835612 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-6wwg6_85b2851b-b372-4935-9b8b-379379b4ae92/insights-operator/1.log" Jul 14 20:34:03.005346 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:03.005314 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhxm7_84af0db3-a613-4183-b841-22d445d83eec/kube-rbac-proxy/0.log" Jul 14 20:34:03.027460 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:03.027431 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhxm7_84af0db3-a613-4183-b841-22d445d83eec/exporter/0.log" Jul 14 20:34:03.054000 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:03.053964 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhxm7_84af0db3-a613-4183-b841-22d445d83eec/extractor/0.log" Jul 14 20:34:03.241817 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:03.241741 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7v52j/perf-node-gather-daemonset-t4vd6" Jul 14 20:34:05.276455 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:05.276423 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lc6qm_99de2193-68ea-4c3f-bf6a-efa88689502a/manager/0.log" Jul 14 20:34:05.531648 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:05.531619 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-fh5pg_f1674e59-0ac4-4f18-8047-4de1c2793908/manager/0.log" Jul 14 20:34:05.702011 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:05.701932 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bcd84_d8a97c62-1a40-4ab4-b6be-25580d851593/server/0.log" Jul 14 20:34:05.811157 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:05.811128 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rgc5q_0fcb2f4e-84a6-4aa3-a585-0c3318a97bf4/manager/0.log" Jul 14 20:34:05.934870 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:05.934843 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-mkd87_fd42ad44-f326-448d-8795-71faf345277d/manager/0.log" Jul 14 20:34:05.957760 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:05.957664 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f59db9768-mkd87_fd42ad44-f326-448d-8795-71faf345277d/copy-manifests/0.log" Jul 14 20:34:06.674844 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:06.674799 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf494s6n_9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c/istio-proxy/0.log" Jul 14 20:34:06.820668 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:06.820635 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5df66fb8c-jh5t7_2e58882e-0431-4140-9998-3303528d1457/kube-auth-proxy/0.log" Jul 14 20:34:06.860817 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:06.860781 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pthzr_a42a78ef-b193-4fc1-8196-2b1979b4a50f/istio-proxy/0.log" Jul 14 20:34:06.924287 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:06.924256 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pthzr_a42a78ef-b193-4fc1-8196-2b1979b4a50f/istio-proxy/1.log" Jul 14 20:34:06.966747 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:06.966658 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-flfq6_a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4/payload-pre-processing/0.log" Jul 14 20:34:06.996473 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:06.996444 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-vjgzw_456b87f2-0077-4159-8049-4eae24c13fd9/payload-processing/0.log" Jul 14 20:34:07.013896 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:07.013865 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b7cc8dd66-pmcdg_acafe8a7-3da1-4bac-b4a9-81aaa72fc21f/router/0.log" Jul 14 20:34:08.239182 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:08.239153 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf494s6n_9ef901b7-0fdb-4cc5-9ec8-4908f555cd2c/istio-proxy/0.log" Jul 14 20:34:08.404158 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:08.404130 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5df66fb8c-jh5t7_2e58882e-0431-4140-9998-3303528d1457/kube-auth-proxy/0.log" Jul 14 20:34:08.443046 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:08.442997 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pthzr_a42a78ef-b193-4fc1-8196-2b1979b4a50f/istio-proxy/0.log" Jul 14 20:34:08.507384 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:08.507294 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pthzr_a42a78ef-b193-4fc1-8196-2b1979b4a50f/istio-proxy/1.log" Jul 14 20:34:08.543866 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:08.543820 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-flfq6_a6cbaa00-d31c-4c98-94bd-ba07e8d1f9b4/payload-pre-processing/0.log" Jul 14 20:34:08.575273 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:08.575243 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-vjgzw_456b87f2-0077-4159-8049-4eae24c13fd9/payload-processing/0.log" Jul 14 20:34:08.589465 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:08.589434 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b7cc8dd66-pmcdg_acafe8a7-3da1-4bac-b4a9-81aaa72fc21f/router/0.log" Jul 14 20:34:12.360384 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:12.360351 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-42h8v_d3850dc3-2e5e-453d-a772-2cbed31dee96/migrator/0.log" Jul 14 20:34:12.381584 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:12.381552 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-42h8v_d3850dc3-2e5e-453d-a772-2cbed31dee96/graceful-termination/0.log" Jul 14 20:34:12.739650 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:12.739550 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-795zv_4c335c34-95b3-4b24-8b1c-0e503876aeb9/kube-storage-version-migrator-operator/1.log" Jul 14 20:34:12.741418 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:12.741398 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-795zv_4c335c34-95b3-4b24-8b1c-0e503876aeb9/kube-storage-version-migrator-operator/0.log" Jul 14 20:34:13.811784 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:13.811747 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6pcwn_9aaae312-ec5a-4cb3-8f65-700e7e980d45/kube-multus/0.log" Jul 14 20:34:13.836846 ip-10-0-133-243 kubenswrapper[2578]: I0714 20:34:13.836812 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gb6f2_fbce3123-7485-48fa-b9fd-3b3e1e43a9ee/kube-multus-additional-cni-plugins/0.log"