Jul 10 19:00:23.127078 ip-10-0-140-210 systemd[1]: Starting Kubernetes Kubelet... Jul 10 19:00:23.564923 ip-10-0-140-210 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:00:23.564923 ip-10-0-140-210 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 19:00:23.565435 ip-10-0-140-210 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:00:23.565435 ip-10-0-140-210 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 19:00:23.565435 ip-10-0-140-210 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:00:23.567191 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.567067 2549 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 19:00:23.573847 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573832 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:23.573847 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573847 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573850 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573854 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573857 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573860 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573863 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573866 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573869 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573871 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573881 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573884 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573887 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573889 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573892 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573894 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573897 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573899 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573901 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573904 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573906 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:23.573914 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573909 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573911 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573914 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573918 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573922 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573925 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573928 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573930 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573933 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573935 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573938 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573941 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573943 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573945 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573948 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573951 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573953 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573956 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573958 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573967 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:23.574394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573969 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573972 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573974 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573976 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573979 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573981 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573984 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573986 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573990 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573993 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573996 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.573999 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574002 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574004 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574008 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574011 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574014 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574017 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574019 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:23.574907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574022 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574025 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574027 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574029 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574032 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574034 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574038 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574040 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574042 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574045 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574047 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574050 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574052 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574061 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574063 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574066 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574068 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574070 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574073 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574075 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:23.575402 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574078 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574080 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574082 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574085 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574087 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574089 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574092 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574096 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574099 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574101 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574104 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574106 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574109 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574112 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574115 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574118 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574120 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574123 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:23.575904 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.574126 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:23.576389 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576376 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:23.576389 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576389 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576394 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576397 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576400 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576403 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576405 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576409 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576412 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576415 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576418 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576420 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576423 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576425 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576428 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576430 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576433 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576435 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576439 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576443 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576445 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:23.576444 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576448 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576451 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576453 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576456 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576459 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576461 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576464 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576466 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576468 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576471 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576473 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576476 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576478 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576480 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576483 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576486 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576488 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576491 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576493 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:23.576995 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576496 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576498 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576501 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576503 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576506 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576508 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576510 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576513 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576515 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576517 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576520 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576522 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576525 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576527 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576529 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576532 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576536 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576540 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576543 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:23.577487 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576546 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576550 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576553 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576556 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576558 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576561 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576563 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576566 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576568 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576570 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576573 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576575 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576578 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576580 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576583 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576585 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576587 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576590 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576592 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:23.577967 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576596 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576599 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576601 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576604 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576607 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576610 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576613 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576615 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576617 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576621 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576625 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576627 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576630 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576632 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576635 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576637 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576640 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576642 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576644 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576647 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:23.578428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.576649 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577085 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577091 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577094 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577098 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577101 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577104 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577107 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577110 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577113 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577116 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577118 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577122 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577124 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577127 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577129 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577132 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577134 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577136 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577139 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:23.578933 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577141 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577144 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577147 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577150 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577153 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577155 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577158 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577160 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577163 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577165 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577168 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577170 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577172 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577175 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577177 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577179 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577183 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577186 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577189 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:23.579428 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577192 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577195 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577197 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577200 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577202 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577204 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577207 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577210 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577212 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577215 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577217 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577220 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577222 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577224 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577230 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577233 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577235 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577238 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577240 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577243 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:23.579916 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577245 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577248 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577250 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577252 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577255 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577266 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577270 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577275 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577279 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577285 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577291 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577294 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577297 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577300 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577303 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577306 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577308 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577312 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577317 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:23.580394 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577320 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577323 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577326 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577329 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577332 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577335 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577337 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577342 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577345 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577347 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577350 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577353 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577356 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577358 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577361 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577363 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577366 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577369 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577371 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577373 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:23.580871 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577376 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.577378 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577454 2549 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577464 2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577474 2549 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577480 2549 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577484 2549 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577487 2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577492 2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577497 2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577500 2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577503 2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577506 2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577509 2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577512 2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577515 2549 flags.go:64] FLAG: --cgroup-root="" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577517 2549 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577520 2549 flags.go:64] FLAG: --client-ca-file="" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577523 2549 flags.go:64] FLAG: --cloud-provider="external" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577527 2549 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577531 2549 flags.go:64] FLAG: --cluster-domain="" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577535 2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577538 2549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577540 2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 19:00:23.581344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577544 2549 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577547 2549 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577550 2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577553 2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577556 2549 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577559 2549 flags.go:64] FLAG: --contention-profiling="false" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577562 2549 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577565 2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577568 2549 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577570 2549 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577574 2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577577 2549 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577580 2549 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577582 2549 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577585 2549 flags.go:64] FLAG: --enable-server="true" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577588 2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577593 2549 flags.go:64] FLAG: --event-burst="100" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577596 2549 flags.go:64] FLAG: --event-qps="50" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577599 2549 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577601 2549 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577604 2549 flags.go:64] FLAG: --eviction-hard="" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577607 2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577610 2549 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577613 2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577616 2549 flags.go:64] FLAG: --eviction-soft="" Jul 10 19:00:23.581933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577618 2549 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577621 2549 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577625 2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577628 2549 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577631 2549 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577633 2549 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577636 2549 flags.go:64] FLAG: --feature-gates="" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577640 2549 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577643 2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577646 2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577648 2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577651 2549 flags.go:64] FLAG: --healthz-port="10248" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577654 2549 flags.go:64] FLAG: --help="false" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577657 2549 flags.go:64] FLAG: --hostname-override="ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577660 2549 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577663 2549 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577665 2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577668 2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577671 2549 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577674 2549 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577677 2549 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577679 2549 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577682 2549 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 19:00:23.582529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577685 2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577688 2549 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577691 2549 flags.go:64] FLAG: --kube-reserved="" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577693 2549 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577696 2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577699 2549 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577702 2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577704 2549 flags.go:64] FLAG: --lock-file="" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577707 2549 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577709 2549 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577712 2549 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577719 2549 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577722 2549 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577725 2549 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577727 2549 flags.go:64] FLAG: --logging-format="text" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577730 2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577734 2549 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577736 2549 flags.go:64] FLAG: --manifest-url="" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577739 2549 flags.go:64] FLAG: --manifest-url-header="" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577743 2549 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577746 2549 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577750 2549 flags.go:64] FLAG: --max-pods="110" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577753 2549 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577756 2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577758 2549 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 19:00:23.583160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577761 2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577764 2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577767 2549 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577770 2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577776 2549 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577779 2549 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577781 2549 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577784 2549 flags.go:64] FLAG: --pod-cidr="" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577787 2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577806 2549 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577810 2549 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577813 2549 flags.go:64] FLAG: --pods-per-core="0" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577815 2549 flags.go:64] FLAG: --port="10250" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577818 2549 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577821 2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03a2c3be78b7afa79" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577824 2549 flags.go:64] FLAG: --qos-reserved="" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577827 2549 flags.go:64] FLAG: --read-only-port="10255" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577829 2549 flags.go:64] FLAG: --register-node="true" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577833 2549 flags.go:64] FLAG: --register-with-taints="" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577837 2549 flags.go:64] FLAG: --registry-burst="10" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577840 2549 flags.go:64] FLAG: --registry-qps="5" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577843 2549 flags.go:64] FLAG: --reserved-cpus="" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577845 2549 flags.go:64] FLAG: --reserved-memory="" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577848 2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577851 2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 19:00:23.583779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577854 2549 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577857 2549 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577859 2549 flags.go:64] FLAG: --runonce="false" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577862 2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577865 2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577868 2549 flags.go:64] FLAG: --seccomp-default="false" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577871 2549 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577873 2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577876 2549 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577879 2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577882 2549 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577884 2549 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577887 2549 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577889 2549 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577892 2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577895 2549 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577898 2549 flags.go:64] FLAG: --system-cgroups="" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577900 2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577906 2549 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577908 2549 flags.go:64] FLAG: --tls-cert-file="" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577911 2549 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577915 2549 flags.go:64] FLAG: --tls-min-version="" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577918 2549 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577920 2549 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577923 2549 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 19:00:23.584527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577927 2549 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577931 2549 flags.go:64] FLAG: --v="2" Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577935 2549 flags.go:64] FLAG: --version="false" Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577939 2549 flags.go:64] FLAG: --vmodule="" Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577942 2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.577946 2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578077 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578082 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578085 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578088 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578090 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578093 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578095 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578098 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578100 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578104 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578106 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578109 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578111 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578113 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578116 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578118 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:23.585397 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578121 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578123 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578125 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578128 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578131 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578134 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578136 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578139 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578141 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578144 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578149 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578154 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578157 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578159 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578162 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578165 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578167 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578170 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578173 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:23.585943 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578175 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578178 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578181 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578183 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578185 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578189 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578193 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578195 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578198 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578200 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578202 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578206 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578208 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578210 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578213 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578215 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578217 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578220 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578223 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:23.586407 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578225 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578229 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578232 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578235 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578239 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578243 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578245 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578248 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578250 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578253 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578255 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578258 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578260 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578263 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578265 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578268 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578270 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578272 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578275 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578277 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:23.586928 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578280 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578282 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578285 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578287 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578289 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578292 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578294 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578297 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578299 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578302 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578304 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578306 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578314 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578316 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578319 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578321 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578325 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578329 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578331 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578333 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:23.587510 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578336 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578338 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578341 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578343 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.578346 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.578350 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.586425 2549 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.586440 2549 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.586459 2549 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.586465 2549 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586521 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586525 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586529 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586533 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586536 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:23.597202 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586539 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586541 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586544 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586547 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586550 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586553 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586555 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586558 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586560 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586563 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586565 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586568 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586570 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586573 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586575 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586578 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586580 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586583 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586585 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586588 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:23.597587 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586590 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586594 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586602 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586606 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586609 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586612 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586615 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586617 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586620 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586622 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586625 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586627 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586630 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586632 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586635 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586637 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586640 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586642 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586644 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586647 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:23.598080 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586649 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586652 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586654 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586657 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586659 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586662 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586664 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586667 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586669 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586672 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586675 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586678 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586680 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586682 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586685 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586688 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586691 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586694 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586697 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586701 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:23.598573 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586704 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586706 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586709 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586711 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586713 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586716 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586718 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586721 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586723 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586725 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586728 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586730 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586733 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586735 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586738 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586740 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586743 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586745 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586748 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:23.599102 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586751 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586753 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586756 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586758 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586761 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586764 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586766 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586768 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586771 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586774 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586776 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586779 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586781 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586785 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:23.599603 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586806 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.586814 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586933 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586939 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586942 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586945 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586948 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586950 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586953 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586956 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586958 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586961 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586964 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586966 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586969 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:23.599968 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586972 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586975 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586978 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586983 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586986 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586989 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586992 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586994 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.586997 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587000 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587003 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587005 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587007 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587010 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587012 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587015 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587018 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587021 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587024 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:23.600344 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587026 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587028 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587031 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587033 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587035 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587038 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587040 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587042 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587045 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587047 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587050 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587052 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587054 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587058 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587060 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587063 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587065 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587068 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587070 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587072 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:23.600824 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587075 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587078 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587081 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587083 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587085 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587088 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587090 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587093 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587096 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587099 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587102 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587105 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587108 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587110 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587112 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587115 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587117 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587119 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587122 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:23.601339 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587124 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587127 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587129 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587132 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587134 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587137 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587139 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587143 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587145 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587148 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587150 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587152 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587155 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587157 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587160 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587162 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587165 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587167 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587169 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587172 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:23.601808 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587174 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587176 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587179 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587181 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587184 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587186 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587188 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:23.587191 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.587195 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.587314 2549 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.598474 2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.600169 2549 server.go:1019] "Starting client certificate rotation" Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.600287 2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 19:00:23.602293 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.600322 2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 19:00:23.603142 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.603128 2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 19:00:23.604273 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.604253 2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 19:00:23.619864 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.619844 2549 log.go:25] "Validated CRI v1 runtime API" Jul 10 19:00:23.622239 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.622225 2549 log.go:25] "Validated CRI v1 image API" Jul 10 19:00:23.623008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.622992 2549 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 19:00:23.625572 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.625555 2549 fs.go:135] Filesystem UUIDs: map[33e03162-e509-453d-8f1f-68427c752ff6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7b0ee09c-72b8-4204-af95-42eb43d4db28:/dev/nvme0n1p4] Jul 10 19:00:23.625619 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.625572 2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 19:00:23.633487 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.633372 2549 manager.go:216] Machine: {Timestamp:2026-07-10 19:00:23.631967995 +0000 UTC m=+0.382872860 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099743 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e769a3259bae6991c467f548aaf83 SystemUUID:ec2e769a-3259-bae6-991c-467f548aaf83 BootID:08c8f355-3032-425c-a0ba-4ec9d961f537 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2c:f8:11:f1:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2c:f8:11:f1:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:ae:8d:96:be:6e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 19:00:23.633487 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.633475 2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 19:00:23.633614 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.633557 2549 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 19:00:23.637139 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.637107 2549 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 19:00:23.637271 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.637143 2549 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 19:00:23.637936 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.637926 2549 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 19:00:23.637967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.637940 2549 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 19:00:23.637967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.637947 2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 19:00:23.637967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.637962 2549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 19:00:23.638046 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.638026 2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 19:00:23.639172 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.639163 2549 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 19:00:23.642832 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.642820 2549 state_mem.go:36] "Initialized new in-memory state store" Jul 10 19:00:23.642944 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.642934 2549 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 19:00:23.642975 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.642964 2549 kubelet.go:486] "Attempting to sync node with API server" Jul 10 19:00:23.642975 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.642974 2549 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 19:00:23.643036 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.642990 2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 19:00:23.643036 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.643022 2549 kubelet.go:398] "Adding apiserver pod source" Jul 10 19:00:23.643036 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.643030 2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 19:00:23.643425 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.643414 2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 19:00:23.643481 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.643433 2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 19:00:23.645838 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.645821 2549 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 19:00:23.647631 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.647618 2549 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 19:00:23.647675 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.647648 2549 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 19:00:23.648486 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648475 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 19:00:23.648518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648491 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 19:00:23.648518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648497 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 19:00:23.648518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648502 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 19:00:23.648518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648510 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 19:00:23.648518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648516 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 19:00:23.648666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648541 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 19:00:23.648666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648547 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 19:00:23.648666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648554 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 19:00:23.648666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648560 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 19:00:23.648666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648569 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 19:00:23.648666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.648577 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 19:00:23.649442 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.649431 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 19:00:23.649442 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.649442 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 19:00:23.652832 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.652818 2549 server.go:1268] "Started kubelet" Jul 10 19:00:23.652929 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.652898 2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 19:00:23.653087 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.653038 2549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 19:00:23.653166 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.653105 2549 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 19:00:23.653672 ip-10-0-140-210 systemd[1]: Started Kubernetes Kubelet. Jul 10 19:00:23.654563 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.654548 2549 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 19:00:23.654756 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.654744 2549 server.go:310] "Adding debug handlers to kubelet server" Jul 10 19:00:23.655588 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.655566 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 19:00:23.655666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.655587 2549 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 19:00:23.655666 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.655609 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 19:00:23.658861 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.658835 2549 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 19:00:23.659666 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.658593 2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-210.ec2.internal.18c102d0e62058ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-210.ec2.internal,UID:ip-10-0-140-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-210.ec2.internal,},FirstTimestamp:2026-07-10 19:00:23.652759724 +0000 UTC m=+0.403664590,LastTimestamp:2026-07-10 19:00:23.652759724 +0000 UTC m=+0.403664590,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-210.ec2.internal,}" Jul 10 19:00:23.660455 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.660438 2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 19:00:23.660547 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.660459 2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 19:00:23.660547 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.660505 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:23.661234 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.661212 2549 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 19:00:23.661315 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.661215 2549 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 19:00:23.661315 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.661300 2549 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 19:00:23.662062 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662047 2549 factory.go:153] Registering CRI-O factory Jul 10 19:00:23.662062 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662062 2549 factory.go:223] Registration of the crio container factory successfully Jul 10 19:00:23.662193 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662128 2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 19:00:23.662193 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662138 2549 factory.go:55] Registering systemd factory Jul 10 19:00:23.662193 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662144 2549 factory.go:223] Registration of the systemd container factory successfully Jul 10 19:00:23.662193 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662160 2549 factory.go:103] Registering Raw factory Jul 10 19:00:23.662193 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662168 2549 manager.go:1196] Started watching for new ooms in manager Jul 10 19:00:23.662366 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662128 2549 reconstruct.go:97] "Volume reconstruction finished" Jul 10 19:00:23.662366 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662213 2549 reconciler.go:29] "Reconciler: start to sync state" Jul 10 19:00:23.662687 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.662671 2549 manager.go:349] Starting recovery of all containers Jul 10 19:00:23.665667 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.665643 2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8b2rt" Jul 10 19:00:23.668916 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.668890 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 19:00:23.669167 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.669138 2549 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 19:00:23.674512 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.674493 2549 manager.go:354] Recovery completed Jul 10 19:00:23.674651 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.674630 2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8b2rt" Jul 10 19:00:23.675779 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.675759 2549 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 10 19:00:23.678672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.678656 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:23.681545 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.681528 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:23.681607 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.681555 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:23.681607 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.681568 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:23.682115 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.682098 2549 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 19:00:23.682115 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.682112 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:23.682267 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.682115 2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 19:00:23.682267 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.682172 2549 state_mem.go:36] "Initialized new in-memory state store" Jul 10 19:00:23.682764 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.682750 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:23.682843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.682776 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:23.682843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.682787 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:23.683176 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.683164 2549 policy_none.go:49] "None policy: Start" Jul 10 19:00:23.683228 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.683183 2549 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 19:00:23.683228 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.683197 2549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 19:00:23.684199 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.684187 2549 policy_none.go:47] "Start" Jul 10 19:00:23.719454 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.719321 2549 manager.go:337] "Starting Device Plugin manager" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.719481 2549 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.719496 2549 server.go:87] "Starting device plugin registration server" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.719695 2549 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.719706 2549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.719849 2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.719921 2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.719928 2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.720475 2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 19:00:23.731658 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.720522 2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:23.796806 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.796752 2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 19:00:23.798060 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.798044 2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 19:00:23.798110 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.798090 2549 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 19:00:23.798143 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.798117 2549 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 19:00:23.798189 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.798155 2549 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 19:00:23.801449 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.801430 2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:23.820016 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.819970 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:23.820782 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.820760 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:23.820886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.820817 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:23.820886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.820841 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:23.820886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.820873 2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.832072 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.832052 2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.832072 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.832072 2549 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-210.ec2.internal\": node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:23.848491 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.848472 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:23.898801 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.898766 2549 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal"] Jul 10 19:00:23.898852 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.898843 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:23.899636 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.899622 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:23.899695 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.899647 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:23.899695 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.899656 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:23.900945 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.900933 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:23.901076 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.901061 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.901126 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.901098 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:23.901622 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.901607 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:23.901622 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.901616 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:23.901728 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.901634 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:23.901728 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.901647 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:23.901728 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.901634 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:23.901839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.901739 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:23.902932 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.902916 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.902980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.902955 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:23.903616 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.903600 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:23.903697 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.903634 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:23.903697 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.903644 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:23.930441 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.930419 2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-210.ec2.internal\" not found" node="ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.934837 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.934820 2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-210.ec2.internal\" not found" node="ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.949212 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:23.949189 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:23.964337 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.964317 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2ad6244da66dd8524b1bacda2ae8771-config\") pod \"kube-apiserver-proxy-ip-10-0-140-210.ec2.internal\" (UID: \"a2ad6244da66dd8524b1bacda2ae8771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.964415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.964339 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87307c9972eaca78fd324aced4761e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal\" (UID: \"87307c9972eaca78fd324aced4761e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:23.964415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:23.964358 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87307c9972eaca78fd324aced4761e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal\" (UID: \"87307c9972eaca78fd324aced4761e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.050163 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.050137 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.064826 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.064788 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2ad6244da66dd8524b1bacda2ae8771-config\") pod \"kube-apiserver-proxy-ip-10-0-140-210.ec2.internal\" (UID: \"a2ad6244da66dd8524b1bacda2ae8771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.064903 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.064832 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87307c9972eaca78fd324aced4761e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal\" (UID: \"87307c9972eaca78fd324aced4761e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.064903 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.064887 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a2ad6244da66dd8524b1bacda2ae8771-config\") pod \"kube-apiserver-proxy-ip-10-0-140-210.ec2.internal\" (UID: \"a2ad6244da66dd8524b1bacda2ae8771\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.065002 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.064929 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/87307c9972eaca78fd324aced4761e70-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal\" (UID: \"87307c9972eaca78fd324aced4761e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.065002 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.064960 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87307c9972eaca78fd324aced4761e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal\" (UID: \"87307c9972eaca78fd324aced4761e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.065002 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.064996 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87307c9972eaca78fd324aced4761e70-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal\" (UID: \"87307c9972eaca78fd324aced4761e70\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.151242 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.151172 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.233683 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.233649 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.238225 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.238012 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:24.251921 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.251902 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.352551 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.352509 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.453070 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.453000 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.553691 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.553660 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.599219 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.599195 2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 19:00:24.599645 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.599355 2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 19:00:24.654772 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.654736 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.679569 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.679540 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:55:23 +0000 UTC" deadline="2027-12-15 07:10:54.34669483 +0000 UTC" Jul 10 19:00:24.679569 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.679568 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12540h10m29.667130962s" Jul 10 19:00:24.755916 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.755850 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.783637 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:24.783608 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87307c9972eaca78fd324aced4761e70.slice/crio-5789f664abb5feed7459d524b297be4f677484e4b7a4efe16458332ac124bc17 WatchSource:0}: Error finding container 5789f664abb5feed7459d524b297be4f677484e4b7a4efe16458332ac124bc17: Status 404 returned error can't find the container with id 5789f664abb5feed7459d524b297be4f677484e4b7a4efe16458332ac124bc17 Jul 10 19:00:24.784072 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:24.784050 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ad6244da66dd8524b1bacda2ae8771.slice/crio-b0e9a0993d5ac59a836d6c0304268faf111e6792a4eccc9c8031a6ff35482420 WatchSource:0}: Error finding container b0e9a0993d5ac59a836d6c0304268faf111e6792a4eccc9c8031a6ff35482420: Status 404 returned error can't find the container with id b0e9a0993d5ac59a836d6c0304268faf111e6792a4eccc9c8031a6ff35482420 Jul 10 19:00:24.788897 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.788882 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:00:24.801712 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.801669 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" event={"ID":"87307c9972eaca78fd324aced4761e70","Type":"ContainerStarted","Data":"5789f664abb5feed7459d524b297be4f677484e4b7a4efe16458332ac124bc17"} Jul 10 19:00:24.802697 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.802676 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" event={"ID":"a2ad6244da66dd8524b1bacda2ae8771","Type":"ContainerStarted","Data":"b0e9a0993d5ac59a836d6c0304268faf111e6792a4eccc9c8031a6ff35482420"} Jul 10 19:00:24.856482 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.856449 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.956967 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:24.956946 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:24.990722 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:24.990705 2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:25.057063 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.056998 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:25.157536 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.157499 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-210.ec2.internal\" not found" Jul 10 19:00:25.248025 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.247993 2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:25.254036 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.254005 2549 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:25.261154 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.261116 2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" Jul 10 19:00:25.272967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.272945 2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 19:00:25.273996 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.273973 2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" Jul 10 19:00:25.288861 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.288844 2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 19:00:25.644113 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.644079 2549 apiserver.go:52] "Watching apiserver" Jul 10 19:00:25.652390 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.652361 2549 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 19:00:25.653575 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.653539 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh","openshift-multus/multus-4zwwt","openshift-multus/multus-additional-cni-plugins-7tvt6","openshift-network-diagnostics/network-check-target-9rcmj","openshift-network-operator/iptables-alerter-5p54z","openshift-cluster-node-tuning-operator/tuned-nvlpc","openshift-dns/node-resolver-gclsw","openshift-image-registry/node-ca-4zngt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal","openshift-multus/network-metrics-daemon-knspp","openshift-ovn-kubernetes/ovnkube-node-cz8np","kube-system/global-pull-secret-syncer-qhkzc","kube-system/konnectivity-agent-mddxb"] Jul 10 19:00:25.656310 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.656285 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.659252 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.659018 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.659252 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.659095 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:25.659423 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.659405 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 19:00:25.659672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.659654 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:00:25.659765 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.659410 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kf7ks\"" Jul 10 19:00:25.661300 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.661277 2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 19:00:25.662096 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.661547 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.664512 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.663835 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.664914 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.664893 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 19:00:25.666500 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.666478 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 19:00:25.666700 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.666680 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 19:00:25.666859 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.666843 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 19:00:25.667235 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.667186 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 19:00:25.667335 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.667256 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zq5fg\"" Jul 10 19:00:25.667457 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.667441 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4688h\"" Jul 10 19:00:25.667667 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.667641 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 19:00:25.667770 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.667708 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:25.667930 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.667785 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:25.669991 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.669974 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.672478 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.672457 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.672946 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.672919 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-modprobe-d\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.673056 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.672952 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysctl-d\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.673056 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.672979 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-systemd\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.673056 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673002 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-host\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.673056 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673027 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-kubelet-config\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.673056 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673051 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-cnibin\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673067 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-cni-multus\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673082 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-kubelet\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673097 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-etc-kubernetes\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673113 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673128 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysctl-conf\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673142 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df823ddd-9e53-491b-9686-e42f2994f50e-tmp\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673156 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-cni-binary-copy\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673171 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-cni-bin\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673187 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-conf-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673202 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673232 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-kubernetes\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673246 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673285 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:00:25.673375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673249 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-dbus\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673418 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673464 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-cni-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673492 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-k8s-cni-cncf-io\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673519 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673542 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-sys\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673552 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673566 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmhb4\" (UniqueName: \"kubernetes.io/projected/df823ddd-9e53-491b-9686-e42f2994f50e-kube-api-access-mmhb4\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673577 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-skhgz\"" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673590 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-system-cni-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673614 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-netns\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673668 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-hostroot\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673695 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-daemon-config\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673719 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-multus-certs\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673744 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8ptt\" (UniqueName: \"kubernetes.io/projected/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-kube-api-access-c8ptt\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673768 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-os-release\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673806 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmwt\" (UniqueName: \"kubernetes.io/projected/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-kube-api-access-5mmwt\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673880 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysconfig\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.674044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673916 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-lib-modules\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.674763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673944 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df823ddd-9e53-491b-9686-e42f2994f50e-etc-tuned\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.674763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673967 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-os-release\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.673984 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-socket-dir-parent\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.674763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.674048 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.674763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.674071 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.674763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.674092 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-run\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.674763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.674119 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-var-lib-kubelet\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.674763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.674141 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cnibin\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.675189 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.675166 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:25.676784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.676760 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gtkvf\"" Jul 10 19:00:25.676866 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.676817 2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 19:00:25.677218 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.677191 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 19:00:25.677396 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.677374 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 19:00:25.677536 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.677257 2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:25.678912 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.678891 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.680956 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.680932 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 19:00:25.681402 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.681340 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 19:00:25.681510 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.681443 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 19:00:25.681510 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.681480 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q6jds\"" Jul 10 19:00:25.681510 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.681489 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.683797 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.683779 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:25.683965 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.683937 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:25.686456 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.686434 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.686934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.686916 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 19:00:25.687826 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.687808 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 19:00:25.687958 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.687938 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 19:00:25.688094 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.688075 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rf7wg\"" Jul 10 19:00:25.688187 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.688148 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 19:00:25.688401 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.688377 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p9chh\"" Jul 10 19:00:25.688718 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.688701 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 19:00:25.695841 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.695677 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 19:00:25.695841 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.695740 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 19:00:25.695841 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.695775 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 19:00:25.697209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.696681 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 19:00:25.697209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.696705 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 19:00:25.697209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.697005 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nbsjf\"" Jul 10 19:00:25.697209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.697021 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 19:00:25.706179 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.706157 2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b9xg9" Jul 10 19:00:25.714605 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.714581 2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b9xg9" Jul 10 19:00:25.763341 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.763312 2549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 19:00:25.774414 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774379 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fce47570-05bd-436a-91d8-9799b73e968d-host\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.774539 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774425 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-kubelet\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.774539 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774449 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysctl-conf\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.774539 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774475 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6qk\" (UniqueName: \"kubernetes.io/projected/92d614fb-0a78-48ad-813b-81cec189119a-kube-api-access-mb6qk\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.774539 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774501 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-systemd\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.774539 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774508 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-kubelet\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.774539 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774525 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e3969ca0-0f62-4628-b2b7-90df227ea65f-iptables-alerter-script\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.774800 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774550 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-conf-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.774800 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774576 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.774800 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774643 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-kubernetes\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.774800 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774684 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-conf-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.774800 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774690 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysctl-conf\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.774800 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774751 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-log-socket\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.774800 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774774 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-cni-netd\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.774800 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774778 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-kubernetes\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774814 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-env-overrides\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774841 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snlpn\" (UniqueName: \"kubernetes.io/projected/e3969ca0-0f62-4628-b2b7-90df227ea65f-kube-api-access-snlpn\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774873 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774894 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-sys\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774911 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmhb4\" (UniqueName: \"kubernetes.io/projected/df823ddd-9e53-491b-9686-e42f2994f50e-kube-api-access-mmhb4\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774927 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-systemd-units\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774931 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-system-cni-dir\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774946 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-netns\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.774974 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:25.775258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775251 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-sys\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775281 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-var-lib-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775308 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-multus-certs\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775336 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775327 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775398 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-run-netns\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775417 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-ovn\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775430 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-netns\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775443 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8ptt\" (UniqueName: \"kubernetes.io/projected/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-kube-api-access-c8ptt\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775491 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysconfig\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775506 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-multus-certs\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775527 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-lib-modules\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775547 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-device-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775586 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysconfig\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775610 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2gmz\" (UniqueName: \"kubernetes.io/projected/c0b5f626-0e73-46e6-8a00-b127ccf339ee-kube-api-access-z2gmz\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775627 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0be705c-d948-4f4e-aa76-5baab302ac9c-agent-certs\") pod \"konnectivity-agent-mddxb\" (UID: \"b0be705c-d948-4f4e-aa76-5baab302ac9c\") " pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775658 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-os-release\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.775755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775687 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775712 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-run\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775769 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-var-lib-kubelet\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775799 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-socket-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775825 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbnj\" (UniqueName: \"kubernetes.io/projected/93b1de2e-aef6-4424-808f-dbecaeff1c47-kube-api-access-cxbnj\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775835 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-run\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775835 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-lib-modules\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775965 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-var-lib-kubelet\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775964 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-os-release\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.775999 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-modprobe-d\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776034 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysctl-d\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776109 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-systemd\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776119 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-modprobe-d\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776179 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-systemd\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776209 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-etc-sysctl-d\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776181 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-kubelet-config\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776278 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776295 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-node-log\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.776550 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776319 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-cnibin\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776315 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-kubelet-config\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776330 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776364 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-cni-multus\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776426 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-cnibin\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776443 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-etc-kubernetes\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776449 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-cni-multus\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776482 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776499 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df823ddd-9e53-491b-9686-e42f2994f50e-tmp\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776568 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-etc-kubernetes\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776596 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-dbus\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776660 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776669 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-sys-fs\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776701 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776728 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-cni-binary-copy\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776744 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-cni-bin\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776749 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-dbus\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.777404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776790 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-var-lib-cni-bin\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776829 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776858 2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776861 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776936 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fce47570-05bd-436a-91d8-9799b73e968d-serviceca\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.776947 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776959 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-cni-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776975 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-k8s-cni-cncf-io\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.776997 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-registration-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.777022 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret podName:2f888fa4-b9d3-45dc-bc95-9801d7ecef40 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:26.276999932 +0000 UTC m=+3.027904794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret") pod "global-pull-secret-syncer-qhkzc" (UID: "2f888fa4-b9d3-45dc-bc95-9801d7ecef40") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777042 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-host-run-k8s-cni-cncf-io\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777047 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-cni-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777055 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-slash\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777084 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777114 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-system-cni-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777139 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovnkube-script-lib\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777160 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3969ca0-0f62-4628-b2b7-90df227ea65f-host-slash\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.778183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777176 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfs4q\" (UniqueName: \"kubernetes.io/projected/fce47570-05bd-436a-91d8-9799b73e968d-kube-api-access-qfs4q\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777180 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-system-cni-dir\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777194 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0be705c-d948-4f4e-aa76-5baab302ac9c-konnectivity-ca\") pod \"konnectivity-agent-mddxb\" (UID: \"b0be705c-d948-4f4e-aa76-5baab302ac9c\") " pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777223 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-hostroot\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777277 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-hostroot\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777297 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-daemon-config\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777328 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b1de2e-aef6-4424-808f-dbecaeff1c47-tmp-dir\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777446 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-os-release\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777471 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmwt\" (UniqueName: \"kubernetes.io/projected/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-kube-api-access-5mmwt\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777491 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df823ddd-9e53-491b-9686-e42f2994f50e-etc-tuned\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777512 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-kubelet\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777534 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovnkube-config\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777556 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovn-node-metrics-cert\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777577 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b1de2e-aef6-4424-808f-dbecaeff1c47-hosts-file\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777597 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-socket-dir-parent\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777618 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777652 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.778877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777676 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-etc-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777701 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cnibin\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777724 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-host\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777749 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8kg\" (UniqueName: \"kubernetes.io/projected/0356a670-6fee-46da-ae40-e88cca1487f3-kube-api-access-jq8kg\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777762 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-os-release\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777774 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-cni-bin\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777850 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cnibin\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777877 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-cni-binary-copy\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777892 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df823ddd-9e53-491b-9686-e42f2994f50e-host\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777897 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-daemon-config\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.777938 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-multus-socket-dir-parent\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.779507 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.778706 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.780908 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.780885 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df823ddd-9e53-491b-9686-e42f2994f50e-tmp\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.780997 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.780956 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/df823ddd-9e53-491b-9686-e42f2994f50e-etc-tuned\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.784161 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.784085 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8ptt\" (UniqueName: \"kubernetes.io/projected/bef4af72-15b8-4dd9-98c1-4ab4ab0608f5-kube-api-access-c8ptt\") pod \"multus-4zwwt\" (UID: \"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5\") " pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.784735 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.784710 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmhb4\" (UniqueName: \"kubernetes.io/projected/df823ddd-9e53-491b-9686-e42f2994f50e-kube-api-access-mmhb4\") pod \"tuned-nvlpc\" (UID: \"df823ddd-9e53-491b-9686-e42f2994f50e\") " pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.790030 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.790011 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmwt\" (UniqueName: \"kubernetes.io/projected/1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4-kube-api-access-5mmwt\") pod \"multus-additional-cni-plugins-7tvt6\" (UID: \"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4\") " pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.878988 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.878960 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-log-socket\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879181 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.878998 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-cni-netd\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879181 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879029 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-env-overrides\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879181 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879099 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-log-socket\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879181 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879101 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-cni-netd\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879181 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879131 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snlpn\" (UniqueName: \"kubernetes.io/projected/e3969ca0-0f62-4628-b2b7-90df227ea65f-kube-api-access-snlpn\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.879181 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879173 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-systemd-units\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879202 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879227 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-var-lib-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879241 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-systemd-units\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879254 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879293 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-run-netns\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879319 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-ovn\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.879333 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879348 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-device-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.879377 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs podName:0356a670-6fee-46da-ae40-e88cca1487f3 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:26.379360337 +0000 UTC m=+3.130265189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs") pod "network-metrics-daemon-knspp" (UID: "0356a670-6fee-46da-ae40-e88cca1487f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879395 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-run-netns\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879395 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-var-lib-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879414 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2gmz\" (UniqueName: \"kubernetes.io/projected/c0b5f626-0e73-46e6-8a00-b127ccf339ee-kube-api-access-z2gmz\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879442 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0be705c-d948-4f4e-aa76-5baab302ac9c-agent-certs\") pod \"konnectivity-agent-mddxb\" (UID: \"b0be705c-d948-4f4e-aa76-5baab302ac9c\") " pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879444 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-ovn\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879486 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-device-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879537 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-env-overrides\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879575 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-socket-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879596 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbnj\" (UniqueName: \"kubernetes.io/projected/93b1de2e-aef6-4424-808f-dbecaeff1c47-kube-api-access-cxbnj\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879617 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879634 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-node-log\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879654 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-sys-fs\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879666 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-socket-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879709 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879669 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879752 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-node-log\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879754 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879762 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879786 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-sys-fs\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.879934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879791 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fce47570-05bd-436a-91d8-9799b73e968d-serviceca\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879811 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-run-ovn-kubernetes\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879897 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-registration-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879925 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-slash\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879953 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.879982 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovnkube-script-lib\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880006 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3969ca0-0f62-4628-b2b7-90df227ea65f-host-slash\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880019 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-slash\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880028 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfs4q\" (UniqueName: \"kubernetes.io/projected/fce47570-05bd-436a-91d8-9799b73e968d-kube-api-access-qfs4q\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880067 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0be705c-d948-4f4e-aa76-5baab302ac9c-konnectivity-ca\") pod \"konnectivity-agent-mddxb\" (UID: \"b0be705c-d948-4f4e-aa76-5baab302ac9c\") " pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880097 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b1de2e-aef6-4424-808f-dbecaeff1c47-tmp-dir\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880124 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-kubelet\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880148 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovnkube-config\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880165 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovn-node-metrics-cert\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880181 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b1de2e-aef6-4424-808f-dbecaeff1c47-hosts-file\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880198 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880205 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fce47570-05bd-436a-91d8-9799b73e968d-serviceca\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.880574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880217 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-etc-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880245 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-etc-openvswitch\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880255 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8kg\" (UniqueName: \"kubernetes.io/projected/0356a670-6fee-46da-ae40-e88cca1487f3-kube-api-access-jq8kg\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880489 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3969ca0-0f62-4628-b2b7-90df227ea65f-host-slash\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880552 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93b1de2e-aef6-4424-808f-dbecaeff1c47-hosts-file\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880789 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovnkube-script-lib\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880797 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovnkube-config\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880799 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0be705c-d948-4f4e-aa76-5baab302ac9c-konnectivity-ca\") pod \"konnectivity-agent-mddxb\" (UID: \"b0be705c-d948-4f4e-aa76-5baab302ac9c\") " pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880840 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-cni-bin\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880878 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880887 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fce47570-05bd-436a-91d8-9799b73e968d-host\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.880917 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881031 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-cni-bin\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881030 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fce47570-05bd-436a-91d8-9799b73e968d-host\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881057 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6qk\" (UniqueName: \"kubernetes.io/projected/92d614fb-0a78-48ad-813b-81cec189119a-kube-api-access-mb6qk\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881059 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93b1de2e-aef6-4424-808f-dbecaeff1c47-tmp-dir\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881081 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/92d614fb-0a78-48ad-813b-81cec189119a-registration-dir\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.881369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881093 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-host-kubelet\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.882281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881193 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-systemd\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.882281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881130 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0b5f626-0e73-46e6-8a00-b127ccf339ee-run-systemd\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.882281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881245 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e3969ca0-0f62-4628-b2b7-90df227ea65f-iptables-alerter-script\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.882281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.881880 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e3969ca0-0f62-4628-b2b7-90df227ea65f-iptables-alerter-script\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.882491 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.882364 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0be705c-d948-4f4e-aa76-5baab302ac9c-agent-certs\") pod \"konnectivity-agent-mddxb\" (UID: \"b0be705c-d948-4f4e-aa76-5baab302ac9c\") " pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:25.883010 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.882993 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0b5f626-0e73-46e6-8a00-b127ccf339ee-ovn-node-metrics-cert\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.891103 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.891084 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:25.891103 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.891105 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:25.891249 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.891118 2549 projected.go:196] Error preparing data for projected volume kube-api-access-2qgxm for pod openshift-network-diagnostics/network-check-target-9rcmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:25.891249 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:25.891219 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm podName:7f8eb8de-8a4d-4081-a196-93d78a90535d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:26.391204668 +0000 UTC m=+3.142109524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2qgxm" (UniqueName: "kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm") pod "network-check-target-9rcmj" (UID: "7f8eb8de-8a4d-4081-a196-93d78a90535d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:25.894758 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.894639 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8kg\" (UniqueName: \"kubernetes.io/projected/0356a670-6fee-46da-ae40-e88cca1487f3-kube-api-access-jq8kg\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:25.895062 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.895037 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfs4q\" (UniqueName: \"kubernetes.io/projected/fce47570-05bd-436a-91d8-9799b73e968d-kube-api-access-qfs4q\") pod \"node-ca-4zngt\" (UID: \"fce47570-05bd-436a-91d8-9799b73e968d\") " pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:25.895154 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.895059 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbnj\" (UniqueName: \"kubernetes.io/projected/93b1de2e-aef6-4424-808f-dbecaeff1c47-kube-api-access-cxbnj\") pod \"node-resolver-gclsw\" (UID: \"93b1de2e-aef6-4424-808f-dbecaeff1c47\") " pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:25.895469 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.895451 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snlpn\" (UniqueName: \"kubernetes.io/projected/e3969ca0-0f62-4628-b2b7-90df227ea65f-kube-api-access-snlpn\") pod \"iptables-alerter-5p54z\" (UID: \"e3969ca0-0f62-4628-b2b7-90df227ea65f\") " pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:25.895545 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.895475 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2gmz\" (UniqueName: \"kubernetes.io/projected/c0b5f626-0e73-46e6-8a00-b127ccf339ee-kube-api-access-z2gmz\") pod \"ovnkube-node-cz8np\" (UID: \"c0b5f626-0e73-46e6-8a00-b127ccf339ee\") " pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:25.896295 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.896278 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6qk\" (UniqueName: \"kubernetes.io/projected/92d614fb-0a78-48ad-813b-81cec189119a-kube-api-access-mb6qk\") pod \"aws-ebs-csi-driver-node-mqqlh\" (UID: \"92d614fb-0a78-48ad-813b-81cec189119a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:25.970951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.970915 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" Jul 10 19:00:25.982133 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.982105 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4zwwt" Jul 10 19:00:25.989985 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.989963 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" Jul 10 19:00:25.998832 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:25.998812 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5p54z" Jul 10 19:00:26.006658 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.006636 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4zngt" Jul 10 19:00:26.013580 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.013557 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:26.020488 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.020468 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" Jul 10 19:00:26.027278 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.027258 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gclsw" Jul 10 19:00:26.033994 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.033976 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:26.284894 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.284788 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:26.285056 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.284962 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:26.285056 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.285040 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret podName:2f888fa4-b9d3-45dc-bc95-9801d7ecef40 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:27.28501993 +0000 UTC m=+4.035924783 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret") pod "global-pull-secret-syncer-qhkzc" (UID: "2f888fa4-b9d3-45dc-bc95-9801d7ecef40") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:26.386099 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.386063 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:26.386286 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.386198 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:26.386286 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.386263 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs podName:0356a670-6fee-46da-ae40-e88cca1487f3 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:27.386242422 +0000 UTC m=+4.137147279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs") pod "network-metrics-daemon-knspp" (UID: "0356a670-6fee-46da-ae40-e88cca1487f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:26.487089 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.487053 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:26.487284 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.487240 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:26.487284 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.487266 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:26.487284 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.487278 2549 projected.go:196] Error preparing data for projected volume kube-api-access-2qgxm for pod openshift-network-diagnostics/network-check-target-9rcmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:26.487425 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.487331 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm podName:7f8eb8de-8a4d-4081-a196-93d78a90535d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:27.487315277 +0000 UTC m=+4.238220130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qgxm" (UniqueName: "kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm") pod "network-check-target-9rcmj" (UID: "7f8eb8de-8a4d-4081-a196-93d78a90535d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:26.718736 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.718572 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:55:25 +0000 UTC" deadline="2028-01-11 16:18:59.481383714 +0000 UTC" Jul 10 19:00:26.718736 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.718671 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13197h18m32.762725365s" Jul 10 19:00:26.799300 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:26.799272 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:26.799452 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:26.799368 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:26.883748 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:26.883706 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a85b4b9_6dba_4db8_8046_8e37cdb1f3a4.slice/crio-3aa1495166c5e4e1e4b503a637c197adddabd02b1165b2675133d5859eb70f0f WatchSource:0}: Error finding container 3aa1495166c5e4e1e4b503a637c197adddabd02b1165b2675133d5859eb70f0f: Status 404 returned error can't find the container with id 3aa1495166c5e4e1e4b503a637c197adddabd02b1165b2675133d5859eb70f0f Jul 10 19:00:26.884457 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:26.884432 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3969ca0_0f62_4628_b2b7_90df227ea65f.slice/crio-7e12531f887e0e472e271445b832bcdd0609c37979a27eac1d7bd9f8e1fa9a66 WatchSource:0}: Error finding container 7e12531f887e0e472e271445b832bcdd0609c37979a27eac1d7bd9f8e1fa9a66: Status 404 returned error can't find the container with id 7e12531f887e0e472e271445b832bcdd0609c37979a27eac1d7bd9f8e1fa9a66 Jul 10 19:00:26.890676 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:26.890657 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf823ddd_9e53_491b_9686_e42f2994f50e.slice/crio-61c4241081cd237b917e070b61c54c21fc3c35454c59e4c1462c45e2a360c3ac WatchSource:0}: Error finding container 61c4241081cd237b917e070b61c54c21fc3c35454c59e4c1462c45e2a360c3ac: Status 404 returned error can't find the container with id 61c4241081cd237b917e070b61c54c21fc3c35454c59e4c1462c45e2a360c3ac Jul 10 19:00:26.891999 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:26.891974 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce47570_05bd_436a_91d8_9799b73e968d.slice/crio-dd89833242185c49d9d2f45c2c93f68e48a57a2447c9c0d1206cfdd1e60f9854 WatchSource:0}: Error finding container dd89833242185c49d9d2f45c2c93f68e48a57a2447c9c0d1206cfdd1e60f9854: Status 404 returned error can't find the container with id dd89833242185c49d9d2f45c2c93f68e48a57a2447c9c0d1206cfdd1e60f9854 Jul 10 19:00:26.892765 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:26.892742 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b1de2e_aef6_4424_808f_dbecaeff1c47.slice/crio-587a2c30de1d52e0408e99b66fba0ba8173658f5906cc5354e162992a1905011 WatchSource:0}: Error finding container 587a2c30de1d52e0408e99b66fba0ba8173658f5906cc5354e162992a1905011: Status 404 returned error can't find the container with id 587a2c30de1d52e0408e99b66fba0ba8173658f5906cc5354e162992a1905011 Jul 10 19:00:26.894178 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:26.894118 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0be705c_d948_4f4e_aa76_5baab302ac9c.slice/crio-4f1cc2269757ca8851cc2c822a5a2f44cc88262de6ba8fdd50597a8d97d7b91e WatchSource:0}: Error finding container 4f1cc2269757ca8851cc2c822a5a2f44cc88262de6ba8fdd50597a8d97d7b91e: Status 404 returned error can't find the container with id 4f1cc2269757ca8851cc2c822a5a2f44cc88262de6ba8fdd50597a8d97d7b91e Jul 10 19:00:26.895831 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:26.895788 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d614fb_0a78_48ad_813b_81cec189119a.slice/crio-4c05f8b22eaf42b80d42c02cb88fb2ba9adef0684e68bd165f7c9011ea9bf352 WatchSource:0}: Error finding container 4c05f8b22eaf42b80d42c02cb88fb2ba9adef0684e68bd165f7c9011ea9bf352: Status 404 returned error can't find the container with id 4c05f8b22eaf42b80d42c02cb88fb2ba9adef0684e68bd165f7c9011ea9bf352 Jul 10 19:00:26.897907 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:00:26.897861 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef4af72_15b8_4dd9_98c1_4ab4ab0608f5.slice/crio-91da527309faf5e59a43709184f2303e051de93b41ce856babbe2ab391131533 WatchSource:0}: Error finding container 91da527309faf5e59a43709184f2303e051de93b41ce856babbe2ab391131533: Status 404 returned error can't find the container with id 91da527309faf5e59a43709184f2303e051de93b41ce856babbe2ab391131533 Jul 10 19:00:27.293386 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.293088 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:27.293386 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.293237 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:27.293386 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.293386 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret podName:2f888fa4-b9d3-45dc-bc95-9801d7ecef40 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:29.293370248 +0000 UTC m=+6.044275106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret") pod "global-pull-secret-syncer-qhkzc" (UID: "2f888fa4-b9d3-45dc-bc95-9801d7ecef40") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:27.395091 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.394414 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:27.395091 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.394616 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:27.395091 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.394685 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs podName:0356a670-6fee-46da-ae40-e88cca1487f3 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:29.394664683 +0000 UTC m=+6.145569541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs") pod "network-metrics-daemon-knspp" (UID: "0356a670-6fee-46da-ae40-e88cca1487f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:27.495813 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.495760 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:27.496057 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.496036 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:27.496147 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.496070 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:27.496147 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.496084 2549 projected.go:196] Error preparing data for projected volume kube-api-access-2qgxm for pod openshift-network-diagnostics/network-check-target-9rcmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:27.496258 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.496176 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm podName:7f8eb8de-8a4d-4081-a196-93d78a90535d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:29.496154075 +0000 UTC m=+6.247058936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qgxm" (UniqueName: "kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm") pod "network-check-target-9rcmj" (UID: "7f8eb8de-8a4d-4081-a196-93d78a90535d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:27.719211 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.719160 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:55:25 +0000 UTC" deadline="2027-12-24 23:43:25.605753263 +0000 UTC" Jul 10 19:00:27.719211 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.719200 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12772h42m57.886558554s" Jul 10 19:00:27.799292 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.799255 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:27.799463 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.799388 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:27.799867 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.799844 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:27.800234 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:27.799980 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:27.814249 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.814039 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gclsw" event={"ID":"93b1de2e-aef6-4424-808f-dbecaeff1c47","Type":"ContainerStarted","Data":"587a2c30de1d52e0408e99b66fba0ba8173658f5906cc5354e162992a1905011"} Jul 10 19:00:27.820675 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.820606 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" event={"ID":"df823ddd-9e53-491b-9686-e42f2994f50e","Type":"ContainerStarted","Data":"61c4241081cd237b917e070b61c54c21fc3c35454c59e4c1462c45e2a360c3ac"} Jul 10 19:00:27.830149 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.830034 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"c9bde91a4135d34c9acdd4afb62004a057328e129732e1af848774abdfdc11e4"} Jul 10 19:00:27.838587 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.838553 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5p54z" event={"ID":"e3969ca0-0f62-4628-b2b7-90df227ea65f","Type":"ContainerStarted","Data":"7e12531f887e0e472e271445b832bcdd0609c37979a27eac1d7bd9f8e1fa9a66"} Jul 10 19:00:27.854081 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.853349 2549 generic.go:356] "Generic (PLEG): container finished" podID="87307c9972eaca78fd324aced4761e70" containerID="8f4ea33d966bd897c2416206510207345d5ebad958ffd5b78778cb7f4ecc6a6d" exitCode=0 Jul 10 19:00:27.854081 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.853435 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" event={"ID":"87307c9972eaca78fd324aced4761e70","Type":"ContainerDied","Data":"8f4ea33d966bd897c2416206510207345d5ebad958ffd5b78778cb7f4ecc6a6d"} Jul 10 19:00:27.863770 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.863710 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4zwwt" event={"ID":"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5","Type":"ContainerStarted","Data":"91da527309faf5e59a43709184f2303e051de93b41ce856babbe2ab391131533"} Jul 10 19:00:27.868067 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.868003 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4zngt" event={"ID":"fce47570-05bd-436a-91d8-9799b73e968d","Type":"ContainerStarted","Data":"dd89833242185c49d9d2f45c2c93f68e48a57a2447c9c0d1206cfdd1e60f9854"} Jul 10 19:00:27.884300 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.884225 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" event={"ID":"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4","Type":"ContainerStarted","Data":"3aa1495166c5e4e1e4b503a637c197adddabd02b1165b2675133d5859eb70f0f"} Jul 10 19:00:27.895001 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.894324 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" event={"ID":"a2ad6244da66dd8524b1bacda2ae8771","Type":"ContainerStarted","Data":"0bb863d519aaa046711e7a96b56c26b2b7adce5bd9ba345b5f73a903f9232bf8"} Jul 10 19:00:27.912500 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.912272 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" event={"ID":"92d614fb-0a78-48ad-813b-81cec189119a","Type":"ContainerStarted","Data":"4c05f8b22eaf42b80d42c02cb88fb2ba9adef0684e68bd165f7c9011ea9bf352"} Jul 10 19:00:27.917499 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:27.917444 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mddxb" event={"ID":"b0be705c-d948-4f4e-aa76-5baab302ac9c","Type":"ContainerStarted","Data":"4f1cc2269757ca8851cc2c822a5a2f44cc88262de6ba8fdd50597a8d97d7b91e"} Jul 10 19:00:28.798663 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:28.798626 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:28.799174 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:28.798771 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:28.937630 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:28.937579 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" event={"ID":"87307c9972eaca78fd324aced4761e70","Type":"ContainerStarted","Data":"0225a32577d3398c3b5b1bebec98fe5f4b5d7301b109ef64101251ec3556a7ae"} Jul 10 19:00:28.958621 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:28.958556 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-210.ec2.internal" podStartSLOduration=3.9585215480000002 podStartE2EDuration="3.958521548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:27.912420108 +0000 UTC m=+4.663324987" watchObservedRunningTime="2026-07-10 19:00:28.958521548 +0000 UTC m=+5.709426424" Jul 10 19:00:28.959554 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:28.959275 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-210.ec2.internal" podStartSLOduration=3.959260966 podStartE2EDuration="3.959260966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:28.958697338 +0000 UTC m=+5.709602203" watchObservedRunningTime="2026-07-10 19:00:28.959260966 +0000 UTC m=+5.710165841" Jul 10 19:00:29.314008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:29.313970 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:29.314199 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.314143 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:29.314258 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.314208 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret podName:2f888fa4-b9d3-45dc-bc95-9801d7ecef40 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:33.314190137 +0000 UTC m=+10.065095010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret") pod "global-pull-secret-syncer-qhkzc" (UID: "2f888fa4-b9d3-45dc-bc95-9801d7ecef40") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:29.414925 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:29.414868 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:29.415104 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.415028 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:29.415104 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.415092 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs podName:0356a670-6fee-46da-ae40-e88cca1487f3 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:33.415071838 +0000 UTC m=+10.165976713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs") pod "network-metrics-daemon-knspp" (UID: "0356a670-6fee-46da-ae40-e88cca1487f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:29.516035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:29.515999 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:29.516241 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.516204 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:29.516241 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.516225 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:29.516241 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.516238 2549 projected.go:196] Error preparing data for projected volume kube-api-access-2qgxm for pod openshift-network-diagnostics/network-check-target-9rcmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:29.516403 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.516297 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm podName:7f8eb8de-8a4d-4081-a196-93d78a90535d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:33.516278693 +0000 UTC m=+10.267183548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qgxm" (UniqueName: "kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm") pod "network-check-target-9rcmj" (UID: "7f8eb8de-8a4d-4081-a196-93d78a90535d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:29.800688 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:29.800614 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:29.801126 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.800740 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:29.801182 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:29.801137 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:29.801258 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:29.801238 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:30.799298 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:30.799268 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:30.799500 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:30.799383 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:31.801982 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:31.801274 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:31.801982 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:31.801397 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:31.801982 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:31.801811 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:31.801982 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:31.801920 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:32.799164 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:32.799128 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:32.799327 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:32.799275 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:33.348270 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:33.348233 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:33.348701 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.348426 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:33.348701 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.348483 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret podName:2f888fa4-b9d3-45dc-bc95-9801d7ecef40 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:41.34846598 +0000 UTC m=+18.099370844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret") pod "global-pull-secret-syncer-qhkzc" (UID: "2f888fa4-b9d3-45dc-bc95-9801d7ecef40") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:33.449541 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:33.449502 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:33.449689 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.449670 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:33.449758 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.449746 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs podName:0356a670-6fee-46da-ae40-e88cca1487f3 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:41.449723244 +0000 UTC m=+18.200628102 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs") pod "network-metrics-daemon-knspp" (UID: "0356a670-6fee-46da-ae40-e88cca1487f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:33.550502 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:33.550472 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:33.550658 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.550634 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:33.550658 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.550655 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:33.550728 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.550669 2549 projected.go:196] Error preparing data for projected volume kube-api-access-2qgxm for pod openshift-network-diagnostics/network-check-target-9rcmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:33.550765 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.550729 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm podName:7f8eb8de-8a4d-4081-a196-93d78a90535d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:41.550710932 +0000 UTC m=+18.301615799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qgxm" (UniqueName: "kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm") pod "network-check-target-9rcmj" (UID: "7f8eb8de-8a4d-4081-a196-93d78a90535d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:33.799806 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:33.799702 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:33.800015 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.799831 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:33.800202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:33.800183 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:33.800302 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:33.800280 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:34.798885 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:34.798852 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:34.799319 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:34.798987 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:35.799930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:35.799377 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:35.799930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:35.799404 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:35.799930 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:35.799527 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:35.799930 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:35.799687 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:36.799163 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:36.798744 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:36.799163 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:36.798889 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:37.798365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:37.798324 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:37.798761 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:37.798339 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:37.798761 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:37.798458 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:37.798761 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:37.798526 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:38.799109 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:38.799076 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:38.799539 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:38.799206 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:38.960439 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:38.960400 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mddxb" event={"ID":"b0be705c-d948-4f4e-aa76-5baab302ac9c","Type":"ContainerStarted","Data":"a7e8ac7fbbb4b49cb4195d0486fb43176db23b2649d83aca9c37d3e96c3adc3e"} Jul 10 19:00:38.988888 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:38.986842 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mddxb" podStartSLOduration=3.92066055 podStartE2EDuration="14.986821301s" totalImagesPullingTime="11.066160751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:38.986556854 +0000 UTC m=+15.737461754" watchObservedRunningTime="2026-07-10 19:00:38.986821301 +0000 UTC m=+15.737726177" Jul 10 19:00:39.802301 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:39.802266 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:39.802709 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:39.802391 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:39.802709 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:39.802462 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:39.802709 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:39.802559 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:40.798545 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:40.798510 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:40.798820 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:40.798638 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:41.410748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:41.410710 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:41.411268 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.410850 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:41.411268 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.410931 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret podName:2f888fa4-b9d3-45dc-bc95-9801d7ecef40 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:57.410910122 +0000 UTC m=+34.161814977 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret") pod "global-pull-secret-syncer-qhkzc" (UID: "2f888fa4-b9d3-45dc-bc95-9801d7ecef40") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:41.511776 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:41.511733 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:41.511952 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.511909 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:41.511999 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.511978 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs podName:0356a670-6fee-46da-ae40-e88cca1487f3 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:57.511957922 +0000 UTC m=+34.262862777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs") pod "network-metrics-daemon-knspp" (UID: "0356a670-6fee-46da-ae40-e88cca1487f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:41.613084 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:41.613049 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:41.613285 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.613234 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:41.613285 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.613259 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:41.613285 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.613272 2549 projected.go:196] Error preparing data for projected volume kube-api-access-2qgxm for pod openshift-network-diagnostics/network-check-target-9rcmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:41.613405 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.613331 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm podName:7f8eb8de-8a4d-4081-a196-93d78a90535d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:57.613316315 +0000 UTC m=+34.364221166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qgxm" (UniqueName: "kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm") pod "network-check-target-9rcmj" (UID: "7f8eb8de-8a4d-4081-a196-93d78a90535d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:41.802235 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:41.802121 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:41.802235 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:41.802198 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:41.802448 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.802292 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:41.802448 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:41.802425 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:42.798608 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:42.798568 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:42.799150 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:42.798704 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:43.801576 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:43.801550 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:43.801576 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:43.801562 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:43.802024 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:43.801659 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:43.802024 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:43.801813 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:43.933276 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:43.933236 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:43.933923 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:43.933899 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:44.799116 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:44.799078 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:44.799320 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:44.799212 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:45.799369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:45.799324 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:45.799870 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:45.799446 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:45.799870 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:45.799496 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:45.799870 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:45.799602 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:46.799440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.799097 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:46.800296 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:46.799560 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:46.978333 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.978264 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" event={"ID":"92d614fb-0a78-48ad-813b-81cec189119a","Type":"ContainerStarted","Data":"977bfe096be6837c9b6b7184fbae5ccf2603f303608a9e9fe738a8b412304fc7"} Jul 10 19:00:46.979541 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.979509 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gclsw" event={"ID":"93b1de2e-aef6-4424-808f-dbecaeff1c47","Type":"ContainerStarted","Data":"dc5880f68f07cafef6e6335db59dc3b1403798cdea57b1ae65f3e7e96ecbec64"} Jul 10 19:00:46.981085 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.981058 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" event={"ID":"df823ddd-9e53-491b-9686-e42f2994f50e","Type":"ContainerStarted","Data":"a046671e65a3ebcd72a16420ed57deed7ed8177f79b82b9488da8ebf569e9fc8"} Jul 10 19:00:46.983713 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.983694 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:00:46.984021 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.984004 2549 generic.go:356] "Generic (PLEG): container finished" podID="c0b5f626-0e73-46e6-8a00-b127ccf339ee" containerID="a18e98b3087a95081da2b5e88dd8b0520232f8ef8a4913548382f6970ce975b9" exitCode=1 Jul 10 19:00:46.984082 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.984068 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"516909f74fb637afa84756f4a031f6ea9058d88ffe0f1c9cee5143fcc678a7e8"} Jul 10 19:00:46.984145 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.984131 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"103ec650ddd30665331be93d57a8919ddc07ecf3310b880ffd418dc74eb20372"} Jul 10 19:00:46.984181 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.984169 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"aa9a9620cb2b01a8731988007bd9f86d313cbd3f3f728d1cb95093be8320df23"} Jul 10 19:00:46.984216 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.984199 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"d9ef0d4e3f6695acfd5a98ac5f02b938cefb049bc5bdf9a4c5bc640da85493ab"} Jul 10 19:00:46.984255 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.984243 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerDied","Data":"a18e98b3087a95081da2b5e88dd8b0520232f8ef8a4913548382f6970ce975b9"} Jul 10 19:00:46.984303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.984291 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"c06fb3e3801d8bd079373ce602df64385f94ab99407ba7e93350e745813a050a"} Jul 10 19:00:46.985309 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.985287 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4zwwt" event={"ID":"bef4af72-15b8-4dd9-98c1-4ab4ab0608f5","Type":"ContainerStarted","Data":"31447137dd5e42aa3ccdff7ce09d10215fc424b7b8676dbdc737bbec14a905f6"} Jul 10 19:00:46.986652 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.986634 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4zngt" event={"ID":"fce47570-05bd-436a-91d8-9799b73e968d","Type":"ContainerStarted","Data":"f7ee17869ba3e8e628b4c2d8458dfabc0e729827ddca791e38244e92b5de3ccc"} Jul 10 19:00:46.988021 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.988001 2549 generic.go:356] "Generic (PLEG): container finished" podID="1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4" containerID="32ef956ace381e7ab6d1d5a86f7ea48ed4254efed73b5f096d3c12ac41a9e47a" exitCode=0 Jul 10 19:00:46.988101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:46.988031 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" event={"ID":"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4","Type":"ContainerDied","Data":"32ef956ace381e7ab6d1d5a86f7ea48ed4254efed73b5f096d3c12ac41a9e47a"} Jul 10 19:00:47.000133 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.000046 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gclsw" podStartSLOduration=11.896262732 podStartE2EDuration="23.00002693s" totalImagesPullingTime="11.103764198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:46.999326934 +0000 UTC m=+23.750231832" watchObservedRunningTime="2026-07-10 19:00:47.00002693 +0000 UTC m=+23.750931826" Jul 10 19:00:47.042248 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.042194 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4zngt" podStartSLOduration=11.942047786 podStartE2EDuration="23.042176553s" totalImagesPullingTime="11.100128767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:47.041606099 +0000 UTC m=+23.792510973" watchObservedRunningTime="2026-07-10 19:00:47.042176553 +0000 UTC m=+23.793081426" Jul 10 19:00:47.061460 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.061401 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nvlpc" podStartSLOduration=3.80151067 podStartE2EDuration="23.061381524s" totalImagesPullingTime="19.259870854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:47.061120014 +0000 UTC m=+23.812024887" watchObservedRunningTime="2026-07-10 19:00:47.061381524 +0000 UTC m=+23.812286399" Jul 10 19:00:47.079380 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.079313 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4zwwt" podStartSLOduration=3.771133283 podStartE2EDuration="23.079295352s" totalImagesPullingTime="19.308162069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:47.079205084 +0000 UTC m=+23.830109988" watchObservedRunningTime="2026-07-10 19:00:47.079295352 +0000 UTC m=+23.830200208" Jul 10 19:00:47.486859 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.486840 2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 19:00:47.739816 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.739616 2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T19:00:47.486855451Z","UUID":"e749ad67-6154-456a-891d-99764bee2bc0","Handler":null,"Name":"","Endpoint":""} Jul 10 19:00:47.741442 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.741418 2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 19:00:47.741636 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.741464 2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 19:00:47.798840 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.798782 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:47.799008 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:47.798931 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:47.799499 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.798788 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:47.799953 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:47.799776 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:47.993089 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.992958 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5p54z" event={"ID":"e3969ca0-0f62-4628-b2b7-90df227ea65f","Type":"ContainerStarted","Data":"ee953ad4a338214c76887bb1e5568421ecd3956975a613f45bc3ff0a9b728b3a"} Jul 10 19:00:47.995895 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:47.995861 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" event={"ID":"92d614fb-0a78-48ad-813b-81cec189119a","Type":"ContainerStarted","Data":"8889d86582e02a7de5941f780310619230315cd65aa87b7781fb1a6c5e0fd4b3"} Jul 10 19:00:48.013828 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:48.013757 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5p54z" podStartSLOduration=12.90692354 podStartE2EDuration="24.013741389s" totalImagesPullingTime="11.106817849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:48.013414458 +0000 UTC m=+24.764319333" watchObservedRunningTime="2026-07-10 19:00:48.013741389 +0000 UTC m=+24.764646262" Jul 10 19:00:48.798739 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:48.798705 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:48.798950 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:48.798838 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:49.001228 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:49.001129 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" event={"ID":"92d614fb-0a78-48ad-813b-81cec189119a","Type":"ContainerStarted","Data":"da884c125a3dc7f79025c2fbd549618859f640e857f8e7629bf363a8a9bf01dd"} Jul 10 19:00:49.021895 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:49.021829 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqqlh" podStartSLOduration=11.751387942000001 podStartE2EDuration="25.021804771s" totalImagesPullingTime="13.270416829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:49.0217062 +0000 UTC m=+25.772611076" watchObservedRunningTime="2026-07-10 19:00:49.021804771 +0000 UTC m=+25.772709642" Jul 10 19:00:49.799096 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:49.798884 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:49.799295 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:49.798978 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:49.799295 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:49.799197 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:49.799295 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:49.799270 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:50.007388 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:50.007359 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:00:50.007904 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:50.007874 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"1db9e534ccdd4dcdb1b7d576a0790e7235fe81071bdbfc75e2ab22d1e0e07989"} Jul 10 19:00:50.218639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:50.218603 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:50.218874 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:50.218805 2549 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 19:00:50.219619 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:50.219596 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mddxb" Jul 10 19:00:50.798897 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:50.798865 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:50.799117 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:50.798987 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:51.799408 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:51.799150 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:51.800172 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:51.799204 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:51.800172 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:51.799590 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:51.800172 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:51.799710 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:52.014948 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:52.014821 2549 generic.go:356] "Generic (PLEG): container finished" podID="1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4" containerID="57c50645191a430cc712c8efeece9bd841145b8812effc543d1d78bdb4bf7cdc" exitCode=0 Jul 10 19:00:52.014948 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:52.014872 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" event={"ID":"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4","Type":"ContainerDied","Data":"57c50645191a430cc712c8efeece9bd841145b8812effc543d1d78bdb4bf7cdc"} Jul 10 19:00:52.018343 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:52.018326 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:00:52.018707 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:52.018685 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"e40e091cf8cbda86608637d8445ad84fa5bffe6f3ab7e0945b68c10b69d9de5e"} Jul 10 19:00:52.019004 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:52.018986 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:52.019173 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:52.019160 2549 scope.go:117] "RemoveContainer" containerID="a18e98b3087a95081da2b5e88dd8b0520232f8ef8a4913548382f6970ce975b9" Jul 10 19:00:52.036534 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:52.036514 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:52.799291 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:52.799262 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:52.799547 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:52.799402 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:53.024289 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.024261 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:00:53.024652 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.024621 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" event={"ID":"c0b5f626-0e73-46e6-8a00-b127ccf339ee","Type":"ContainerStarted","Data":"3881d60d1db798111260d83baadafdfc85ced7476d2f4b4c56ccec92f8f7a9ba"} Jul 10 19:00:53.024848 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.024829 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:53.024920 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.024907 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:53.041854 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.041828 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:00:53.064421 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.064296 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" podStartSLOduration=9.730255664 podStartE2EDuration="29.064278723s" totalImagesPullingTime="19.334023059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:53.063180717 +0000 UTC m=+29.814085601" watchObservedRunningTime="2026-07-10 19:00:53.064278723 +0000 UTC m=+29.815183594" Jul 10 19:00:53.454873 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.454756 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mddxb_b0be705c-d948-4f4e-aa76-5baab302ac9c/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:53.559315 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.559281 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mddxb_b0be705c-d948-4f4e-aa76-5baab302ac9c/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:53.756035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.755954 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-knspp"] Jul 10 19:00:53.756195 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.756098 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:53.756257 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:53.756221 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:53.756554 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.756531 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9rcmj"] Jul 10 19:00:53.756670 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.756616 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:53.756727 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:53.756692 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:53.762345 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.762325 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qhkzc"] Jul 10 19:00:53.762442 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:53.762429 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:53.762552 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:53.762525 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:54.029548 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:54.029454 2549 generic.go:356] "Generic (PLEG): container finished" podID="1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4" containerID="fc7c93f1d28d731bc5d417497f054269c2fc4914f7d1051528b3d6375dde96d3" exitCode=0 Jul 10 19:00:54.029976 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:54.029541 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" event={"ID":"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4","Type":"ContainerDied","Data":"fc7c93f1d28d731bc5d417497f054269c2fc4914f7d1051528b3d6375dde96d3"} Jul 10 19:00:55.238905 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:55.238649 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mddxb_b0be705c-d948-4f4e-aa76-5baab302ac9c/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:55.307901 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:55.307853 2549 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mddxb_b0be705c-d948-4f4e-aa76-5baab302ac9c/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:55.799023 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:55.798987 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:55.799023 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:55.798995 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:55.799260 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:55.799158 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:55.799260 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:55.799213 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:55.799362 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:55.799325 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:55.799410 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:55.799400 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:56.036722 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:56.036630 2549 generic.go:356] "Generic (PLEG): container finished" podID="1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4" containerID="314701920cf63d7720deec2d054224b3e773eecf425d9453d2aa7274d05abfaa" exitCode=0 Jul 10 19:00:56.036722 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:56.036692 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" event={"ID":"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4","Type":"ContainerDied","Data":"314701920cf63d7720deec2d054224b3e773eecf425d9453d2aa7274d05abfaa"} Jul 10 19:00:57.429162 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:57.429067 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:57.429971 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.429220 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:57.429971 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.429296 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret podName:2f888fa4-b9d3-45dc-bc95-9801d7ecef40 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:29.429276571 +0000 UTC m=+66.180181438 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret") pod "global-pull-secret-syncer-qhkzc" (UID: "2f888fa4-b9d3-45dc-bc95-9801d7ecef40") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:57.530357 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:57.530319 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:57.530554 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.530496 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:57.530625 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.530575 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs podName:0356a670-6fee-46da-ae40-e88cca1487f3 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:29.530554339 +0000 UTC m=+66.281459196 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs") pod "network-metrics-daemon-knspp" (UID: "0356a670-6fee-46da-ae40-e88cca1487f3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:57.631446 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:57.631403 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:57.631618 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.631582 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:57.631618 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.631605 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:57.631618 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.631618 2549 projected.go:196] Error preparing data for projected volume kube-api-access-2qgxm for pod openshift-network-diagnostics/network-check-target-9rcmj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:57.631727 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.631678 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm podName:7f8eb8de-8a4d-4081-a196-93d78a90535d nodeName:}" failed. No retries permitted until 2026-07-10 19:01:29.631663678 +0000 UTC m=+66.382568533 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2qgxm" (UniqueName: "kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm") pod "network-check-target-9rcmj" (UID: "7f8eb8de-8a4d-4081-a196-93d78a90535d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:57.799398 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:57.799305 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:57.799578 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:57.799309 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:57.799578 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.799460 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhkzc" podUID="2f888fa4-b9d3-45dc-bc95-9801d7ecef40" Jul 10 19:00:57.799578 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:57.799305 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:57.799732 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.799581 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-knspp" podUID="0356a670-6fee-46da-ae40-e88cca1487f3" Jul 10 19:00:57.799732 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:57.799703 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9rcmj" podUID="7f8eb8de-8a4d-4081-a196-93d78a90535d" Jul 10 19:00:59.103521 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.103492 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-210.ec2.internal" event="NodeReady" Jul 10 19:00:59.104083 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.103655 2549 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 19:00:59.157574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.157541 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ksmpt"] Jul 10 19:00:59.163946 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.163912 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5bcjs"] Jul 10 19:00:59.164102 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.164036 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.166777 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.166746 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gmlfz\"" Jul 10 19:00:59.166777 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.166761 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 19:00:59.167001 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.166926 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 19:00:59.167328 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.167308 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5bcjs" Jul 10 19:00:59.169770 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.169749 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jcvlx\"" Jul 10 19:00:59.169925 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.169900 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 19:00:59.170030 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.169933 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 19:00:59.170249 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.170233 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 19:00:59.173039 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.173018 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ksmpt"] Jul 10 19:00:59.175565 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.175543 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5bcjs"] Jul 10 19:00:59.244503 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.244468 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nmx\" (UniqueName: \"kubernetes.io/projected/deff8a3a-41de-4b18-8f3e-a29843bab383-kube-api-access-45nmx\") pod \"ingress-canary-5bcjs\" (UID: \"deff8a3a-41de-4b18-8f3e-a29843bab383\") " pod="openshift-ingress-canary/ingress-canary-5bcjs" Jul 10 19:00:59.244503 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.244512 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deff8a3a-41de-4b18-8f3e-a29843bab383-cert\") pod \"ingress-canary-5bcjs\" (UID: \"deff8a3a-41de-4b18-8f3e-a29843bab383\") " pod="openshift-ingress-canary/ingress-canary-5bcjs" Jul 10 19:00:59.244767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.244547 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.244767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.244626 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrk4\" (UniqueName: \"kubernetes.io/projected/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-kube-api-access-znrk4\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.244767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.244658 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-tmp-dir\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.244767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.244712 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-config-volume\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.345839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.345785 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nmx\" (UniqueName: \"kubernetes.io/projected/deff8a3a-41de-4b18-8f3e-a29843bab383-kube-api-access-45nmx\") pod \"ingress-canary-5bcjs\" (UID: \"deff8a3a-41de-4b18-8f3e-a29843bab383\") " pod="openshift-ingress-canary/ingress-canary-5bcjs" Jul 10 19:00:59.346053 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.345850 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deff8a3a-41de-4b18-8f3e-a29843bab383-cert\") pod \"ingress-canary-5bcjs\" (UID: \"deff8a3a-41de-4b18-8f3e-a29843bab383\") " pod="openshift-ingress-canary/ingress-canary-5bcjs" Jul 10 19:00:59.346053 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.345893 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.346053 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:59.346013 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:00:59.346217 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.346040 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrk4\" (UniqueName: \"kubernetes.io/projected/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-kube-api-access-znrk4\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.346217 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:59.346074 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls podName:a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:59.84605433 +0000 UTC m=+36.596959183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls") pod "dns-default-ksmpt" (UID: "a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186") : secret "dns-default-metrics-tls" not found Jul 10 19:00:59.346217 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.346107 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-tmp-dir\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.346217 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.346167 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-config-volume\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.346833 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.346670 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-tmp-dir\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.346833 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.346808 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-config-volume\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.351135 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.351098 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/deff8a3a-41de-4b18-8f3e-a29843bab383-cert\") pod \"ingress-canary-5bcjs\" (UID: \"deff8a3a-41de-4b18-8f3e-a29843bab383\") " pod="openshift-ingress-canary/ingress-canary-5bcjs" Jul 10 19:00:59.362243 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.362185 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nmx\" (UniqueName: \"kubernetes.io/projected/deff8a3a-41de-4b18-8f3e-a29843bab383-kube-api-access-45nmx\") pod \"ingress-canary-5bcjs\" (UID: \"deff8a3a-41de-4b18-8f3e-a29843bab383\") " pod="openshift-ingress-canary/ingress-canary-5bcjs" Jul 10 19:00:59.381196 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.381161 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrk4\" (UniqueName: \"kubernetes.io/projected/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-kube-api-access-znrk4\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.484651 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.484612 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5bcjs" Jul 10 19:00:59.798778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.798688 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:00:59.798778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.798712 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:00:59.799034 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.798698 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:00:59.801916 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.801894 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zb6nn\"" Jul 10 19:00:59.801916 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.801912 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 19:00:59.802888 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.802864 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 19:00:59.802888 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.802881 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 19:00:59.803154 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.803140 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqhv9\"" Jul 10 19:00:59.803340 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.803324 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 19:00:59.849700 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:00:59.849663 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:00:59.849893 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:59.849872 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:00:59.849955 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:00:59.849941 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls podName:a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:00.84992486 +0000 UTC m=+37.600829716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls") pod "dns-default-ksmpt" (UID: "a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186") : secret "dns-default-metrics-tls" not found Jul 10 19:01:00.240473 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:00.240306 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5bcjs"] Jul 10 19:01:00.243926 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:00.243900 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeff8a3a_41de_4b18_8f3e_a29843bab383.slice/crio-f6176b13664b85118f0ff5a4c6a8ebb58eb51ec047825420995f2569284ac7ec WatchSource:0}: Error finding container f6176b13664b85118f0ff5a4c6a8ebb58eb51ec047825420995f2569284ac7ec: Status 404 returned error can't find the container with id f6176b13664b85118f0ff5a4c6a8ebb58eb51ec047825420995f2569284ac7ec Jul 10 19:01:00.858410 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:00.858361 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:00.858612 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:00.858547 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:00.858675 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:00.858635 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls podName:a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:02.858610789 +0000 UTC m=+39.609515670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls") pod "dns-default-ksmpt" (UID: "a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186") : secret "dns-default-metrics-tls" not found Jul 10 19:01:01.054610 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.054567 2549 generic.go:356] "Generic (PLEG): container finished" podID="1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4" containerID="42f7ab28c8a237be9436a879f399b130e37f93195cba9844bcbcadae5ad684dd" exitCode=0 Jul 10 19:01:01.054778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.054652 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" event={"ID":"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4","Type":"ContainerDied","Data":"42f7ab28c8a237be9436a879f399b130e37f93195cba9844bcbcadae5ad684dd"} Jul 10 19:01:01.055901 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.055868 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5bcjs" event={"ID":"deff8a3a-41de-4b18-8f3e-a29843bab383","Type":"ContainerStarted","Data":"f6176b13664b85118f0ff5a4c6a8ebb58eb51ec047825420995f2569284ac7ec"} Jul 10 19:01:01.236455 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.236419 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8b6wq"] Jul 10 19:01:01.240170 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.240148 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.243595 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.243573 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 19:01:01.243977 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.243594 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f67rl\"" Jul 10 19:01:01.244035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.244009 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 19:01:01.244217 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.244199 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 19:01:01.244316 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.244301 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 19:01:01.255123 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.255093 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8b6wq"] Jul 10 19:01:01.362422 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.362385 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.362624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.362452 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae7dfa1d-acb5-4192-91f8-241e98c91f61-crio-socket\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.362624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.362517 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.362727 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.362640 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae7dfa1d-acb5-4192-91f8-241e98c91f61-data-volume\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.362727 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.362679 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85wc\" (UniqueName: \"kubernetes.io/projected/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-api-access-q85wc\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.463352 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.463245 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae7dfa1d-acb5-4192-91f8-241e98c91f61-data-volume\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.463352 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.463300 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85wc\" (UniqueName: \"kubernetes.io/projected/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-api-access-q85wc\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.463352 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.463338 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.463640 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:01.463473 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:01.463640 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:01.463536 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:01.963518143 +0000 UTC m=+38.714423011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:01.463640 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.463550 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae7dfa1d-acb5-4192-91f8-241e98c91f61-crio-socket\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.463640 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.463569 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.463887 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.463664 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae7dfa1d-acb5-4192-91f8-241e98c91f61-data-volume\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.463887 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:01.463702 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:01:01.463887 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:01.463749 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:01.963738695 +0000 UTC m=+38.714643579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : configmap "kube-rbac-proxy" not found Jul 10 19:01:01.463887 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.463832 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae7dfa1d-acb5-4192-91f8-241e98c91f61-crio-socket\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.478916 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.478886 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85wc\" (UniqueName: \"kubernetes.io/projected/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-api-access-q85wc\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.967914 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.967868 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.968090 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:01.967966 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:01.968090 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:01.968043 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:01:01.968090 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:01.968070 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:01.968229 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:01.968124 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:02.968106055 +0000 UTC m=+39.719010908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : configmap "kube-rbac-proxy" not found Jul 10 19:01:01.968229 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:01.968140 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:02.96813367 +0000 UTC m=+39.719038522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:02.044039 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.044008 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv"] Jul 10 19:01:02.049995 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.049965 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx"] Jul 10 19:01:02.050154 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.049967 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv" Jul 10 19:01:02.052721 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.052694 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:01:02.052721 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.052716 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 19:01:02.052908 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.052701 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-78zjj\"" Jul 10 19:01:02.054130 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.054108 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:02.056441 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.056416 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv"] Jul 10 19:01:02.057520 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.057502 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 19:01:02.057647 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.057625 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 19:01:02.057767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.057736 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:01:02.057767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.057757 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7q2gt\"" Jul 10 19:01:02.069865 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.069444 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx"] Jul 10 19:01:02.072895 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.072836 2549 generic.go:356] "Generic (PLEG): container finished" podID="1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4" containerID="94cd77eb7dd88d9f02041e010eedbe8942dd439bb4389a52d51353692bf1a772" exitCode=0 Jul 10 19:01:02.072994 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.072904 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" event={"ID":"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4","Type":"ContainerDied","Data":"94cd77eb7dd88d9f02041e010eedbe8942dd439bb4389a52d51353692bf1a772"} Jul 10 19:01:02.170073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.169785 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:02.170265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.170171 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls9px\" (UniqueName: \"kubernetes.io/projected/50137b74-0aab-4bc8-9568-06486fe76427-kube-api-access-ls9px\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:02.170265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.170223 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6v2j\" (UniqueName: \"kubernetes.io/projected/92a9c9d5-5f40-445d-9337-c8216a134c1d-kube-api-access-v6v2j\") pod \"volume-data-source-validator-78495597f6-p59jv\" (UID: \"92a9c9d5-5f40-445d-9337-c8216a134c1d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv" Jul 10 19:01:02.270785 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.270688 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:02.270785 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.270771 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls9px\" (UniqueName: \"kubernetes.io/projected/50137b74-0aab-4bc8-9568-06486fe76427-kube-api-access-ls9px\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:02.271338 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.270843 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6v2j\" (UniqueName: \"kubernetes.io/projected/92a9c9d5-5f40-445d-9337-c8216a134c1d-kube-api-access-v6v2j\") pod \"volume-data-source-validator-78495597f6-p59jv\" (UID: \"92a9c9d5-5f40-445d-9337-c8216a134c1d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv" Jul 10 19:01:02.271338 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.270887 2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 19:01:02.271338 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.270962 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls podName:50137b74-0aab-4bc8-9568-06486fe76427 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:02.77094242 +0000 UTC m=+39.521847272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls") pod "cluster-samples-operator-65db858894-mgrlx" (UID: "50137b74-0aab-4bc8-9568-06486fe76427") : secret "samples-operator-tls" not found Jul 10 19:01:02.281513 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.281479 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls9px\" (UniqueName: \"kubernetes.io/projected/50137b74-0aab-4bc8-9568-06486fe76427-kube-api-access-ls9px\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:02.281684 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.281492 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6v2j\" (UniqueName: \"kubernetes.io/projected/92a9c9d5-5f40-445d-9337-c8216a134c1d-kube-api-access-v6v2j\") pod \"volume-data-source-validator-78495597f6-p59jv\" (UID: \"92a9c9d5-5f40-445d-9337-c8216a134c1d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv" Jul 10 19:01:02.366868 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.366832 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv" Jul 10 19:01:02.621438 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.621408 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gclsw_93b1de2e-aef6-4424-808f-dbecaeff1c47/dns-node-resolver/0.log" Jul 10 19:01:02.665486 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.665457 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv"] Jul 10 19:01:02.757590 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:02.757561 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a9c9d5_5f40_445d_9337_c8216a134c1d.slice/crio-88bf7c7e8345678c8e1dda2b91c945f3c01e162b93625ce2a15716974f166c10 WatchSource:0}: Error finding container 88bf7c7e8345678c8e1dda2b91c945f3c01e162b93625ce2a15716974f166c10: Status 404 returned error can't find the container with id 88bf7c7e8345678c8e1dda2b91c945f3c01e162b93625ce2a15716974f166c10 Jul 10 19:01:02.776520 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.776488 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:02.776664 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.776655 2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 19:01:02.776731 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.776720 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls podName:50137b74-0aab-4bc8-9568-06486fe76427 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:03.776701866 +0000 UTC m=+40.527606722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls") pod "cluster-samples-operator-65db858894-mgrlx" (UID: "50137b74-0aab-4bc8-9568-06486fe76427") : secret "samples-operator-tls" not found Jul 10 19:01:02.877764 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.877732 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:02.877918 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.877893 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:02.877965 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.877953 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls podName:a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:06.87793722 +0000 UTC m=+43.628842089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls") pod "dns-default-ksmpt" (UID: "a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186") : secret "dns-default-metrics-tls" not found Jul 10 19:01:02.978624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.978588 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:02.978832 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:02.978654 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:02.978832 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.978728 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:01:02.978832 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.978755 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:02.978937 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.978837 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:04.97881393 +0000 UTC m=+41.729718804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:02.978937 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:02.978855 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:04.978845654 +0000 UTC m=+41.729750506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : configmap "kube-rbac-proxy" not found Jul 10 19:01:03.074000 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.073957 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-kpccd"] Jul 10 19:01:03.089528 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.089482 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5bcjs" event={"ID":"deff8a3a-41de-4b18-8f3e-a29843bab383","Type":"ContainerStarted","Data":"d28071437c366f05ca5fc2bc65c3d2f9d3069dd52cc80fe0f2d0ba2f062fa19c"} Jul 10 19:01:03.089528 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.089499 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.089766 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.089558 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-kpccd"] Jul 10 19:01:03.089766 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.089587 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" event={"ID":"1a85b4b9-6dba-4db8-8046-8e37cdb1f3a4","Type":"ContainerStarted","Data":"184477a9d7649f03e749146ac07b5f669e464814afb0c6b8230be63d551c4626"} Jul 10 19:01:03.089766 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.089609 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv" event={"ID":"92a9c9d5-5f40-445d-9337-c8216a134c1d","Type":"ContainerStarted","Data":"88bf7c7e8345678c8e1dda2b91c945f3c01e162b93625ce2a15716974f166c10"} Jul 10 19:01:03.092257 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.092231 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 10 19:01:03.092387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.092317 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 10 19:01:03.092463 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.092231 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:01:03.092463 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.092397 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 10 19:01:03.092586 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.092531 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-gv9mq\"" Jul 10 19:01:03.098781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.098756 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 10 19:01:03.098781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.098746 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5bcjs" podStartSLOduration=1.5588696789999998 podStartE2EDuration="4.098726114s" totalImagesPullingTime="2.539856435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:03.098099477 +0000 UTC m=+39.849004351" watchObservedRunningTime="2026-07-10 19:01:03.098726114 +0000 UTC m=+39.849630989" Jul 10 19:01:03.136674 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.136617 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7tvt6" podStartSLOduration=17.41523171 podStartE2EDuration="39.136600307s" totalImagesPullingTime="21.721368597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:03.135656121 +0000 UTC m=+39.886561019" watchObservedRunningTime="2026-07-10 19:01:03.136600307 +0000 UTC m=+39.887505282" Jul 10 19:01:03.179572 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.179542 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjb8\" (UniqueName: \"kubernetes.io/projected/b7d9cb01-a594-425b-ab20-68088641139a-kube-api-access-zrjb8\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.179703 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.179576 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9cb01-a594-425b-ab20-68088641139a-trusted-ca\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.179755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.179738 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d9cb01-a594-425b-ab20-68088641139a-config\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.179858 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.179838 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d9cb01-a594-425b-ab20-68088641139a-serving-cert\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.280963 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.280916 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d9cb01-a594-425b-ab20-68088641139a-config\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.281963 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.281010 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d9cb01-a594-425b-ab20-68088641139a-serving-cert\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.281963 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.281080 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjb8\" (UniqueName: \"kubernetes.io/projected/b7d9cb01-a594-425b-ab20-68088641139a-kube-api-access-zrjb8\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.281963 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.281107 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9cb01-a594-425b-ab20-68088641139a-trusted-ca\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.282187 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.282158 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d9cb01-a594-425b-ab20-68088641139a-config\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.282729 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.282518 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9cb01-a594-425b-ab20-68088641139a-trusted-ca\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.286613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.286574 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d9cb01-a594-425b-ab20-68088641139a-serving-cert\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.292435 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.292408 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjb8\" (UniqueName: \"kubernetes.io/projected/b7d9cb01-a594-425b-ab20-68088641139a-kube-api-access-zrjb8\") pod \"console-operator-7866497494-kpccd\" (UID: \"b7d9cb01-a594-425b-ab20-68088641139a\") " pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.400898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.400773 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:03.421402 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.421377 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4zngt_fce47570-05bd-436a-91d8-9799b73e968d/node-ca/0.log" Jul 10 19:01:03.558413 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.558377 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-kpccd"] Jul 10 19:01:03.562073 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:03.562040 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9cb01_a594_425b_ab20_68088641139a.slice/crio-273a4fbd76c2f94d0b941cecda89abd23aa91900cd76f6532d6d4e67c7536624 WatchSource:0}: Error finding container 273a4fbd76c2f94d0b941cecda89abd23aa91900cd76f6532d6d4e67c7536624: Status 404 returned error can't find the container with id 273a4fbd76c2f94d0b941cecda89abd23aa91900cd76f6532d6d4e67c7536624 Jul 10 19:01:03.786160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:03.786080 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:03.786319 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:03.786196 2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 19:01:03.786319 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:03.786250 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls podName:50137b74-0aab-4bc8-9568-06486fe76427 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:05.78623576 +0000 UTC m=+42.537140615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls") pod "cluster-samples-operator-65db858894-mgrlx" (UID: "50137b74-0aab-4bc8-9568-06486fe76427") : secret "samples-operator-tls" not found Jul 10 19:01:04.085324 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:04.085293 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kpccd" event={"ID":"b7d9cb01-a594-425b-ab20-68088641139a","Type":"ContainerStarted","Data":"273a4fbd76c2f94d0b941cecda89abd23aa91900cd76f6532d6d4e67c7536624"} Jul 10 19:01:04.421742 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:04.421717 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5bcjs_deff8a3a-41de-4b18-8f3e-a29843bab383/serve-healthcheck-canary/0.log" Jul 10 19:01:04.997449 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:04.997407 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:04.997654 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:04.997492 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:04.997654 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:04.997544 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:04.997654 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:04.997582 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:01:04.997654 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:04.997624 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:08.997604778 +0000 UTC m=+45.748509629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:04.997654 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:04.997642 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:08.997634027 +0000 UTC m=+45.748538884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : configmap "kube-rbac-proxy" not found Jul 10 19:01:05.089087 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:05.089048 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv" event={"ID":"92a9c9d5-5f40-445d-9337-c8216a134c1d","Type":"ContainerStarted","Data":"2b281289ad04a5abf008b4b5e827b969d09bab8b3ffc10a3747a9c24972d9e10"} Jul 10 19:01:05.106425 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:05.106376 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p59jv" podStartSLOduration=1.548156631 podStartE2EDuration="3.106360617s" totalImagesPullingTime="1.558203986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:05.105831128 +0000 UTC m=+41.856736025" watchObservedRunningTime="2026-07-10 19:01:05.106360617 +0000 UTC m=+41.857265490" Jul 10 19:01:05.802586 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:05.802547 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:05.803084 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:05.802823 2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 19:01:05.803084 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:05.802886 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls podName:50137b74-0aab-4bc8-9568-06486fe76427 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:09.8028716 +0000 UTC m=+46.553776454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls") pod "cluster-samples-operator-65db858894-mgrlx" (UID: "50137b74-0aab-4bc8-9568-06486fe76427") : secret "samples-operator-tls" not found Jul 10 19:01:06.913038 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:06.912996 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:06.913409 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:06.913143 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:06.913409 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:06.913202 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls podName:a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:14.913187483 +0000 UTC m=+51.664092334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls") pod "dns-default-ksmpt" (UID: "a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186") : secret "dns-default-metrics-tls" not found Jul 10 19:01:07.070288 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.067548 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8"] Jul 10 19:01:07.084409 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.084379 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8"] Jul 10 19:01:07.084582 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.084468 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.086995 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.086966 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 19:01:07.086995 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.086966 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:01:07.087182 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.087053 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 19:01:07.088106 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.088085 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nqrbw\"" Jul 10 19:01:07.088196 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.088115 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 19:01:07.095911 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.095894 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/0.log" Jul 10 19:01:07.096002 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.095927 2549 generic.go:356] "Generic (PLEG): container finished" podID="b7d9cb01-a594-425b-ab20-68088641139a" containerID="13fe8d4076b8d7b20022eec42ffe49072d1e2bb3aaae74f9641f122094b3e537" exitCode=255 Jul 10 19:01:07.096002 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.095954 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kpccd" event={"ID":"b7d9cb01-a594-425b-ab20-68088641139a","Type":"ContainerDied","Data":"13fe8d4076b8d7b20022eec42ffe49072d1e2bb3aaae74f9641f122094b3e537"} Jul 10 19:01:07.096163 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.096150 2549 scope.go:117] "RemoveContainer" containerID="13fe8d4076b8d7b20022eec42ffe49072d1e2bb3aaae74f9641f122094b3e537" Jul 10 19:01:07.175559 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.175481 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7"] Jul 10 19:01:07.189644 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.189616 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.190242 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.190185 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7"] Jul 10 19:01:07.193063 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.192786 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 19:01:07.193063 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.192878 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sffdc\"" Jul 10 19:01:07.193215 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.193142 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 19:01:07.193215 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.193193 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:01:07.193403 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.193376 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 19:01:07.215860 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.215808 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f20e4509-f5be-402b-bf6f-1178cbbff0f2-serving-cert\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.216058 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.216031 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f20e4509-f5be-402b-bf6f-1178cbbff0f2-config\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.216116 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.216073 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv669\" (UniqueName: \"kubernetes.io/projected/f20e4509-f5be-402b-bf6f-1178cbbff0f2-kube-api-access-dv669\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.317514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.317480 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f20e4509-f5be-402b-bf6f-1178cbbff0f2-config\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.317514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.317518 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv669\" (UniqueName: \"kubernetes.io/projected/f20e4509-f5be-402b-bf6f-1178cbbff0f2-kube-api-access-dv669\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.317767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.317589 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f20e4509-f5be-402b-bf6f-1178cbbff0f2-serving-cert\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.317767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.317616 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c63a7aa-df11-4c44-bdc5-e19b075bf311-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.317767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.317654 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c63a7aa-df11-4c44-bdc5-e19b075bf311-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.317767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.317705 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwl9\" (UniqueName: \"kubernetes.io/projected/2c63a7aa-df11-4c44-bdc5-e19b075bf311-kube-api-access-llwl9\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.318097 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.318079 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f20e4509-f5be-402b-bf6f-1178cbbff0f2-config\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.321008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.320987 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f20e4509-f5be-402b-bf6f-1178cbbff0f2-serving-cert\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.327155 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.327131 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv669\" (UniqueName: \"kubernetes.io/projected/f20e4509-f5be-402b-bf6f-1178cbbff0f2-kube-api-access-dv669\") pod \"service-ca-operator-7695fc6c8c-6lmf8\" (UID: \"f20e4509-f5be-402b-bf6f-1178cbbff0f2\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.393201 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.393164 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" Jul 10 19:01:07.418610 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.418581 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c63a7aa-df11-4c44-bdc5-e19b075bf311-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.418749 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.418616 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwl9\" (UniqueName: \"kubernetes.io/projected/2c63a7aa-df11-4c44-bdc5-e19b075bf311-kube-api-access-llwl9\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.418832 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.418751 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c63a7aa-df11-4c44-bdc5-e19b075bf311-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.419420 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.419346 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2c63a7aa-df11-4c44-bdc5-e19b075bf311-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.421112 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.421088 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c63a7aa-df11-4c44-bdc5-e19b075bf311-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.428534 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.428478 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwl9\" (UniqueName: \"kubernetes.io/projected/2c63a7aa-df11-4c44-bdc5-e19b075bf311-kube-api-access-llwl9\") pod \"kube-storage-version-migrator-operator-799cb7f958-7g2d7\" (UID: \"2c63a7aa-df11-4c44-bdc5-e19b075bf311\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.500125 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.500092 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" Jul 10 19:01:07.535085 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.535060 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8"] Jul 10 19:01:07.538193 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:07.538163 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20e4509_f5be_402b_bf6f_1178cbbff0f2.slice/crio-251b0733f29bad1019abaf920d6d023de9218a55935d41dfa12f4b280df1e3b9 WatchSource:0}: Error finding container 251b0733f29bad1019abaf920d6d023de9218a55935d41dfa12f4b280df1e3b9: Status 404 returned error can't find the container with id 251b0733f29bad1019abaf920d6d023de9218a55935d41dfa12f4b280df1e3b9 Jul 10 19:01:07.656877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:07.656845 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7"] Jul 10 19:01:07.659367 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:07.659341 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c63a7aa_df11_4c44_bdc5_e19b075bf311.slice/crio-82131b7d10d9ab515bf12843250c3acb344fe91eb9e28b2d12ae664d0f2d647b WatchSource:0}: Error finding container 82131b7d10d9ab515bf12843250c3acb344fe91eb9e28b2d12ae664d0f2d647b: Status 404 returned error can't find the container with id 82131b7d10d9ab515bf12843250c3acb344fe91eb9e28b2d12ae664d0f2d647b Jul 10 19:01:08.101353 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:08.101324 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:01:08.101871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:08.101851 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/0.log" Jul 10 19:01:08.101922 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:08.101901 2549 generic.go:356] "Generic (PLEG): container finished" podID="b7d9cb01-a594-425b-ab20-68088641139a" containerID="4d2fea557e2a1a19aa739bfd7abdd76676ec1bcc703328380f5371740e62b6b0" exitCode=255 Jul 10 19:01:08.102024 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:08.101999 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kpccd" event={"ID":"b7d9cb01-a594-425b-ab20-68088641139a","Type":"ContainerDied","Data":"4d2fea557e2a1a19aa739bfd7abdd76676ec1bcc703328380f5371740e62b6b0"} Jul 10 19:01:08.102073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:08.102061 2549 scope.go:117] "RemoveContainer" containerID="13fe8d4076b8d7b20022eec42ffe49072d1e2bb3aaae74f9641f122094b3e537" Jul 10 19:01:08.102310 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:08.102289 2549 scope.go:117] "RemoveContainer" containerID="4d2fea557e2a1a19aa739bfd7abdd76676ec1bcc703328380f5371740e62b6b0" Jul 10 19:01:08.102507 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:08.102484 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kpccd_openshift-console-operator(b7d9cb01-a594-425b-ab20-68088641139a)\"" pod="openshift-console-operator/console-operator-7866497494-kpccd" podUID="b7d9cb01-a594-425b-ab20-68088641139a" Jul 10 19:01:08.103996 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:08.103968 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" event={"ID":"f20e4509-f5be-402b-bf6f-1178cbbff0f2","Type":"ContainerStarted","Data":"251b0733f29bad1019abaf920d6d023de9218a55935d41dfa12f4b280df1e3b9"} Jul 10 19:01:08.105418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:08.105398 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" event={"ID":"2c63a7aa-df11-4c44-bdc5-e19b075bf311","Type":"ContainerStarted","Data":"82131b7d10d9ab515bf12843250c3acb344fe91eb9e28b2d12ae664d0f2d647b"} Jul 10 19:01:09.033583 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:09.033550 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:09.033843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:09.033632 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:09.033843 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:09.033688 2549 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:01:09.033843 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:09.033730 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:09.033843 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:09.033754 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:17.033738976 +0000 UTC m=+53.784643828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : configmap "kube-rbac-proxy" not found Jul 10 19:01:09.033843 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:09.033772 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:17.033761634 +0000 UTC m=+53.784666486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:09.109736 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:09.109700 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:01:09.110217 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:09.110129 2549 scope.go:117] "RemoveContainer" containerID="4d2fea557e2a1a19aa739bfd7abdd76676ec1bcc703328380f5371740e62b6b0" Jul 10 19:01:09.110413 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:09.110325 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kpccd_openshift-console-operator(b7d9cb01-a594-425b-ab20-68088641139a)\"" pod="openshift-console-operator/console-operator-7866497494-kpccd" podUID="b7d9cb01-a594-425b-ab20-68088641139a" Jul 10 19:01:09.840610 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:09.840567 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:09.840826 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:09.840752 2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 19:01:09.840907 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:09.840880 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls podName:50137b74-0aab-4bc8-9568-06486fe76427 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:17.840857195 +0000 UTC m=+54.591762065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls") pod "cluster-samples-operator-65db858894-mgrlx" (UID: "50137b74-0aab-4bc8-9568-06486fe76427") : secret "samples-operator-tls" not found Jul 10 19:01:11.116477 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:11.116445 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" event={"ID":"f20e4509-f5be-402b-bf6f-1178cbbff0f2","Type":"ContainerStarted","Data":"740a4f606c48a260229b0638d709f4b0cfa828da13e748e340004a1905519ce0"} Jul 10 19:01:12.121609 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:12.121575 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" event={"ID":"2c63a7aa-df11-4c44-bdc5-e19b075bf311","Type":"ContainerStarted","Data":"ac1afd3a7b7f6955dda4f3af3d852701fc029b81c536e682376c07736afea0ce"} Jul 10 19:01:12.144536 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:12.144460 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" podStartSLOduration=1.801727685 podStartE2EDuration="5.144444088s" totalImagesPullingTime="3.342716403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:12.143702742 +0000 UTC m=+48.894607611" watchObservedRunningTime="2026-07-10 19:01:12.144444088 +0000 UTC m=+48.895348961" Jul 10 19:01:12.166831 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:12.166761 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" podStartSLOduration=1.707093362 podStartE2EDuration="5.166749875s" totalImagesPullingTime="3.459656513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:12.166736748 +0000 UTC m=+48.917641651" watchObservedRunningTime="2026-07-10 19:01:12.166749875 +0000 UTC m=+48.917654775" Jul 10 19:01:13.400990 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:13.400955 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:13.401371 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:13.401015 2549 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:13.401371 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:13.401332 2549 scope.go:117] "RemoveContainer" containerID="4d2fea557e2a1a19aa739bfd7abdd76676ec1bcc703328380f5371740e62b6b0" Jul 10 19:01:13.401488 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:13.401467 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-kpccd_openshift-console-operator(b7d9cb01-a594-425b-ab20-68088641139a)\"" pod="openshift-console-operator/console-operator-7866497494-kpccd" podUID="b7d9cb01-a594-425b-ab20-68088641139a" Jul 10 19:01:14.984169 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:14.984130 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:14.984553 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:14.984284 2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:14.984553 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:14.984347 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls podName:a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:30.984331408 +0000 UTC m=+67.735236259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls") pod "dns-default-ksmpt" (UID: "a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186") : secret "dns-default-metrics-tls" not found Jul 10 19:01:17.099068 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:17.099035 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:17.099512 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:17.099095 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:17.099512 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:17.099191 2549 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:17.099512 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:17.099264 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls podName:ae7dfa1d-acb5-4192-91f8-241e98c91f61 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:33.09924614 +0000 UTC m=+69.850150993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8b6wq" (UID: "ae7dfa1d-acb5-4192-91f8-241e98c91f61") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:17.099666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:17.099541 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae7dfa1d-acb5-4192-91f8-241e98c91f61-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:17.905354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:17.905323 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:17.905530 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:17.905477 2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 19:01:17.905578 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:17.905532 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls podName:50137b74-0aab-4bc8-9568-06486fe76427 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:33.905519469 +0000 UTC m=+70.656424325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls") pod "cluster-samples-operator-65db858894-mgrlx" (UID: "50137b74-0aab-4bc8-9568-06486fe76427") : secret "samples-operator-tls" not found Jul 10 19:01:25.050358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:25.050332 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cz8np" Jul 10 19:01:26.799216 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:26.799189 2549 scope.go:117] "RemoveContainer" containerID="4d2fea557e2a1a19aa739bfd7abdd76676ec1bcc703328380f5371740e62b6b0" Jul 10 19:01:27.168810 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:27.168779 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:01:27.168967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:27.168844 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-kpccd" event={"ID":"b7d9cb01-a594-425b-ab20-68088641139a","Type":"ContainerStarted","Data":"ff7db5c7437155ef4f9590d35c142637255752996ce2621c902966e028e359dd"} Jul 10 19:01:27.169118 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:27.169097 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:27.245778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:27.245749 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-kpccd" Jul 10 19:01:27.270186 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:27.270125 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-kpccd" podStartSLOduration=21.598840709 podStartE2EDuration="24.270103557s" totalImagesPullingTime="2.671262848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:27.188180797 +0000 UTC m=+63.939085700" watchObservedRunningTime="2026-07-10 19:01:27.270103557 +0000 UTC m=+64.021008430" Jul 10 19:01:29.491982 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.491935 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:01:29.494615 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.494593 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 19:01:29.504399 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.504379 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2f888fa4-b9d3-45dc-bc95-9801d7ecef40-original-pull-secret\") pod \"global-pull-secret-syncer-qhkzc\" (UID: \"2f888fa4-b9d3-45dc-bc95-9801d7ecef40\") " pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:01:29.512335 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.512318 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhkzc" Jul 10 19:01:29.593459 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.593413 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:01:29.596359 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.596337 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 19:01:29.606713 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.606693 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0356a670-6fee-46da-ae40-e88cca1487f3-metrics-certs\") pod \"network-metrics-daemon-knspp\" (UID: \"0356a670-6fee-46da-ae40-e88cca1487f3\") " pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:01:29.640123 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.640093 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qhkzc"] Jul 10 19:01:29.643211 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:29.643187 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f888fa4_b9d3_45dc_bc95_9801d7ecef40.slice/crio-942974cd250ee9fce2b330a2aedc1a03afed0f55c136dd63925ea9edfc22453d WatchSource:0}: Error finding container 942974cd250ee9fce2b330a2aedc1a03afed0f55c136dd63925ea9edfc22453d: Status 404 returned error can't find the container with id 942974cd250ee9fce2b330a2aedc1a03afed0f55c136dd63925ea9edfc22453d Jul 10 19:01:29.694546 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.694515 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:01:29.697201 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.697186 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 19:01:29.708195 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.708175 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 19:01:29.718664 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.718637 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qgxm\" (UniqueName: \"kubernetes.io/projected/7f8eb8de-8a4d-4081-a196-93d78a90535d-kube-api-access-2qgxm\") pod \"network-check-target-9rcmj\" (UID: \"7f8eb8de-8a4d-4081-a196-93d78a90535d\") " pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:01:29.821909 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.821885 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zb6nn\"" Jul 10 19:01:29.827848 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.827827 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqhv9\"" Jul 10 19:01:29.829546 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.829528 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-knspp" Jul 10 19:01:29.836959 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.836942 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:01:29.959854 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.959823 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-knspp"] Jul 10 19:01:29.962342 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:29.962318 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0356a670_6fee_46da_ae40_e88cca1487f3.slice/crio-817c28fd3f6ba8a513756e71591eec375f769a79d7361e51af087aac9c0f30e9 WatchSource:0}: Error finding container 817c28fd3f6ba8a513756e71591eec375f769a79d7361e51af087aac9c0f30e9: Status 404 returned error can't find the container with id 817c28fd3f6ba8a513756e71591eec375f769a79d7361e51af087aac9c0f30e9 Jul 10 19:01:29.975066 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:29.975043 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9rcmj"] Jul 10 19:01:29.977886 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:29.977863 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8eb8de_8a4d_4081_a196_93d78a90535d.slice/crio-b6e053eeb26afefaf5fec07cc33728bb5e028115113d4769e1869ef918fea9f4 WatchSource:0}: Error finding container b6e053eeb26afefaf5fec07cc33728bb5e028115113d4769e1869ef918fea9f4: Status 404 returned error can't find the container with id b6e053eeb26afefaf5fec07cc33728bb5e028115113d4769e1869ef918fea9f4 Jul 10 19:01:30.180631 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:30.180548 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qhkzc" event={"ID":"2f888fa4-b9d3-45dc-bc95-9801d7ecef40","Type":"ContainerStarted","Data":"942974cd250ee9fce2b330a2aedc1a03afed0f55c136dd63925ea9edfc22453d"} Jul 10 19:01:30.181868 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:30.181830 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9rcmj" event={"ID":"7f8eb8de-8a4d-4081-a196-93d78a90535d","Type":"ContainerStarted","Data":"b6e053eeb26afefaf5fec07cc33728bb5e028115113d4769e1869ef918fea9f4"} Jul 10 19:01:30.183397 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:30.183366 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knspp" event={"ID":"0356a670-6fee-46da-ae40-e88cca1487f3","Type":"ContainerStarted","Data":"817c28fd3f6ba8a513756e71591eec375f769a79d7361e51af087aac9c0f30e9"} Jul 10 19:01:31.009236 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:31.009193 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:31.013299 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:31.013277 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186-metrics-tls\") pod \"dns-default-ksmpt\" (UID: \"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186\") " pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:31.281381 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:31.281174 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gmlfz\"" Jul 10 19:01:31.289095 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:31.289070 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:31.496517 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:31.496485 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ksmpt"] Jul 10 19:01:31.500008 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:31.499981 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ca9b75_92b4_4cf5_b2c9_218d1c4f2186.slice/crio-505aa4586fab3df9ec2670f8921bc0032263c5a58c7ab58059a6caa574930626 WatchSource:0}: Error finding container 505aa4586fab3df9ec2670f8921bc0032263c5a58c7ab58059a6caa574930626: Status 404 returned error can't find the container with id 505aa4586fab3df9ec2670f8921bc0032263c5a58c7ab58059a6caa574930626 Jul 10 19:01:32.194101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:32.194056 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knspp" event={"ID":"0356a670-6fee-46da-ae40-e88cca1487f3","Type":"ContainerStarted","Data":"56fbafb0e779ad087344b1a47d1dc6d44821d425fad73eed166c1d837e68ec92"} Jul 10 19:01:32.194578 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:32.194132 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-knspp" event={"ID":"0356a670-6fee-46da-ae40-e88cca1487f3","Type":"ContainerStarted","Data":"2353fa7ebcba0864364114a2217d960ba8be3aeba22a062beb9480e20cd2bf3a"} Jul 10 19:01:32.195899 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:32.195864 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ksmpt" event={"ID":"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186","Type":"ContainerStarted","Data":"505aa4586fab3df9ec2670f8921bc0032263c5a58c7ab58059a6caa574930626"} Jul 10 19:01:32.226445 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:32.226385 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-knspp" podStartSLOduration=67.089094786 podStartE2EDuration="1m8.226369452s" totalImagesPullingTime="1.137274666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:32.226190769 +0000 UTC m=+68.977095647" watchObservedRunningTime="2026-07-10 19:01:32.226369452 +0000 UTC m=+68.977274325" Jul 10 19:01:33.126361 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:33.126322 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:33.129103 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:33.129073 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae7dfa1d-acb5-4192-91f8-241e98c91f61-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8b6wq\" (UID: \"ae7dfa1d-acb5-4192-91f8-241e98c91f61\") " pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:33.353754 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:33.353724 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f67rl\"" Jul 10 19:01:33.360867 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:33.360847 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8b6wq" Jul 10 19:01:33.932446 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:33.932407 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:33.935311 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:33.935281 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/50137b74-0aab-4bc8-9568-06486fe76427-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-mgrlx\" (UID: \"50137b74-0aab-4bc8-9568-06486fe76427\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:34.178635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.178594 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7q2gt\"" Jul 10 19:01:34.186365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.186289 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" Jul 10 19:01:34.415112 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.415076 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c"] Jul 10 19:01:34.416968 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.416947 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-99qsw"] Jul 10 19:01:34.417130 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.417080 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.418744 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.418722 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" Jul 10 19:01:34.422461 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.422434 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 19:01:34.422574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.422515 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 10 19:01:34.422830 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.422782 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 19:01:34.423354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.423256 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kcfvn\"" Jul 10 19:01:34.423354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.423296 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 19:01:34.423625 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.423599 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 10 19:01:34.424159 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.424141 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 19:01:34.436415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.436366 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-99qsw"] Jul 10 19:01:34.437502 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.437480 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e85f93bd-0c5e-403f-b443-9cc24790e26c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-99qsw\" (UID: \"e85f93bd-0c5e-403f-b443-9cc24790e26c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" Jul 10 19:01:34.437624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.437517 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8b2f8b-d0fe-4263-865b-2a839e95789a-tmp\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.437624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.437549 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8f8b2f8b-d0fe-4263-865b-2a839e95789a-klusterlet-config\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.437624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.437574 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e85f93bd-0c5e-403f-b443-9cc24790e26c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-99qsw\" (UID: \"e85f93bd-0c5e-403f-b443-9cc24790e26c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" Jul 10 19:01:34.437811 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.437654 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xrr\" (UniqueName: \"kubernetes.io/projected/8f8b2f8b-d0fe-4263-865b-2a839e95789a-kube-api-access-d4xrr\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.439158 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.439140 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c"] Jul 10 19:01:34.491910 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.491878 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf"] Jul 10 19:01:34.494204 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.494173 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" Jul 10 19:01:34.504612 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.504160 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kt2fz\"" Jul 10 19:01:34.504612 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.504459 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 19:01:34.516474 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.516444 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf"] Jul 10 19:01:34.538467 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.538436 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b50a5fd9-7250-458b-80c0-24b1fa7e35a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d6f6d66-tgwrf\" (UID: \"b50a5fd9-7250-458b-80c0-24b1fa7e35a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" Jul 10 19:01:34.538630 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.538496 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e85f93bd-0c5e-403f-b443-9cc24790e26c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-99qsw\" (UID: \"e85f93bd-0c5e-403f-b443-9cc24790e26c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" Jul 10 19:01:34.538630 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.538533 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7p4\" (UniqueName: \"kubernetes.io/projected/b50a5fd9-7250-458b-80c0-24b1fa7e35a3-kube-api-access-7q7p4\") pod \"managed-serviceaccount-addon-agent-6d6f6d66-tgwrf\" (UID: \"b50a5fd9-7250-458b-80c0-24b1fa7e35a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" Jul 10 19:01:34.538630 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.538565 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8b2f8b-d0fe-4263-865b-2a839e95789a-tmp\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.538630 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.538592 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8f8b2f8b-d0fe-4263-865b-2a839e95789a-klusterlet-config\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.538630 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.538608 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e85f93bd-0c5e-403f-b443-9cc24790e26c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-99qsw\" (UID: \"e85f93bd-0c5e-403f-b443-9cc24790e26c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" Jul 10 19:01:34.538932 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.538636 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xrr\" (UniqueName: \"kubernetes.io/projected/8f8b2f8b-d0fe-4263-865b-2a839e95789a-kube-api-access-d4xrr\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.539601 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.539576 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8b2f8b-d0fe-4263-865b-2a839e95789a-tmp\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.540285 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.540233 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e85f93bd-0c5e-403f-b443-9cc24790e26c-nginx-conf\") pod \"networking-console-plugin-6cf856758c-99qsw\" (UID: \"e85f93bd-0c5e-403f-b443-9cc24790e26c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" Jul 10 19:01:34.541813 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.541774 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e85f93bd-0c5e-403f-b443-9cc24790e26c-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-99qsw\" (UID: \"e85f93bd-0c5e-403f-b443-9cc24790e26c\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" Jul 10 19:01:34.542202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.542182 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8f8b2f8b-d0fe-4263-865b-2a839e95789a-klusterlet-config\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.596321 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.596281 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xrr\" (UniqueName: \"kubernetes.io/projected/8f8b2f8b-d0fe-4263-865b-2a839e95789a-kube-api-access-d4xrr\") pod \"klusterlet-addon-workmgr-565c6f4f58-qnk6c\" (UID: \"8f8b2f8b-d0fe-4263-865b-2a839e95789a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.639979 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.639945 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b50a5fd9-7250-458b-80c0-24b1fa7e35a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d6f6d66-tgwrf\" (UID: \"b50a5fd9-7250-458b-80c0-24b1fa7e35a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" Jul 10 19:01:34.640143 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.640012 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7p4\" (UniqueName: \"kubernetes.io/projected/b50a5fd9-7250-458b-80c0-24b1fa7e35a3-kube-api-access-7q7p4\") pod \"managed-serviceaccount-addon-agent-6d6f6d66-tgwrf\" (UID: \"b50a5fd9-7250-458b-80c0-24b1fa7e35a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" Jul 10 19:01:34.642714 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.642680 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b50a5fd9-7250-458b-80c0-24b1fa7e35a3-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d6f6d66-tgwrf\" (UID: \"b50a5fd9-7250-458b-80c0-24b1fa7e35a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" Jul 10 19:01:34.658245 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.658212 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7p4\" (UniqueName: \"kubernetes.io/projected/b50a5fd9-7250-458b-80c0-24b1fa7e35a3-kube-api-access-7q7p4\") pod \"managed-serviceaccount-addon-agent-6d6f6d66-tgwrf\" (UID: \"b50a5fd9-7250-458b-80c0-24b1fa7e35a3\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" Jul 10 19:01:34.728913 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.728819 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c87d7859f-vvbzf"] Jul 10 19:01:34.730147 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.730113 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:34.731548 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.731519 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.734184 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.734165 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 19:01:34.734738 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.734715 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 19:01:34.734984 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.734961 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" Jul 10 19:01:34.735412 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.735390 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 19:01:34.735622 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.735602 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r62lv\"" Jul 10 19:01:34.741281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.741257 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 19:01:34.762978 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.762949 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c87d7859f-vvbzf"] Jul 10 19:01:34.818373 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.818334 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" Jul 10 19:01:34.842515 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.842479 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83f8db00-c888-4ce8-8637-bb43dc72920e-image-registry-private-configuration\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.842845 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.842533 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83f8db00-c888-4ce8-8637-bb43dc72920e-trusted-ca\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.842845 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.842570 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-registry-tls\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.842845 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.842597 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-bound-sa-token\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.842845 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.842627 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83f8db00-c888-4ce8-8637-bb43dc72920e-registry-certificates\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.842845 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.842662 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83f8db00-c888-4ce8-8637-bb43dc72920e-installation-pull-secrets\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.842845 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.842688 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmz5f\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-kube-api-access-pmz5f\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.842845 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.842728 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83f8db00-c888-4ce8-8637-bb43dc72920e-ca-trust-extracted\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.943686 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.943654 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83f8db00-c888-4ce8-8637-bb43dc72920e-image-registry-private-configuration\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.943898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.943694 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83f8db00-c888-4ce8-8637-bb43dc72920e-trusted-ca\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.943898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.943729 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-registry-tls\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.943898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.943755 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-bound-sa-token\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.943898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.943785 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83f8db00-c888-4ce8-8637-bb43dc72920e-registry-certificates\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.943898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.943847 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83f8db00-c888-4ce8-8637-bb43dc72920e-installation-pull-secrets\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.943898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.943871 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmz5f\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-kube-api-access-pmz5f\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.944138 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.943923 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83f8db00-c888-4ce8-8637-bb43dc72920e-ca-trust-extracted\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.944966 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.944444 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/83f8db00-c888-4ce8-8637-bb43dc72920e-ca-trust-extracted\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.945088 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.945020 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/83f8db00-c888-4ce8-8637-bb43dc72920e-registry-certificates\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.945349 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.945301 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/83f8db00-c888-4ce8-8637-bb43dc72920e-trusted-ca\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.946678 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.946657 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/83f8db00-c888-4ce8-8637-bb43dc72920e-image-registry-private-configuration\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.946903 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.946885 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-registry-tls\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.947239 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.947215 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/83f8db00-c888-4ce8-8637-bb43dc72920e-installation-pull-secrets\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.962169 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.962146 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-bound-sa-token\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:34.964486 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:34.964466 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmz5f\" (UniqueName: \"kubernetes.io/projected/83f8db00-c888-4ce8-8637-bb43dc72920e-kube-api-access-pmz5f\") pod \"image-registry-5c87d7859f-vvbzf\" (UID: \"83f8db00-c888-4ce8-8637-bb43dc72920e\") " pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:35.043731 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:35.043649 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:35.569499 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:35.569059 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8b6wq"] Jul 10 19:01:35.579384 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:35.579302 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7dfa1d_acb5_4192_91f8_241e98c91f61.slice/crio-a16771f6bf667218cc0e46c72a4fcf9992fa7afe58d68b7353c3a692b8a4fbc0 WatchSource:0}: Error finding container a16771f6bf667218cc0e46c72a4fcf9992fa7afe58d68b7353c3a692b8a4fbc0: Status 404 returned error can't find the container with id a16771f6bf667218cc0e46c72a4fcf9992fa7afe58d68b7353c3a692b8a4fbc0 Jul 10 19:01:35.597187 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:35.597123 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf"] Jul 10 19:01:35.599969 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:35.599691 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb50a5fd9_7250_458b_80c0_24b1fa7e35a3.slice/crio-642cd88e1c2f5d8c2000ea5517f70585db6937cc11c65cd3decb02bc3aeb2d0c WatchSource:0}: Error finding container 642cd88e1c2f5d8c2000ea5517f70585db6937cc11c65cd3decb02bc3aeb2d0c: Status 404 returned error can't find the container with id 642cd88e1c2f5d8c2000ea5517f70585db6937cc11c65cd3decb02bc3aeb2d0c Jul 10 19:01:35.611932 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:35.611607 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx"] Jul 10 19:01:35.807568 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:35.807492 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-99qsw"] Jul 10 19:01:35.810219 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:35.810196 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85f93bd_0c5e_403f_b443_9cc24790e26c.slice/crio-b66e26387b803d858b8c10cff95ba9e47d6917112762e757b726d31831589cc0 WatchSource:0}: Error finding container b66e26387b803d858b8c10cff95ba9e47d6917112762e757b726d31831589cc0: Status 404 returned error can't find the container with id b66e26387b803d858b8c10cff95ba9e47d6917112762e757b726d31831589cc0 Jul 10 19:01:35.853739 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:35.853717 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c"] Jul 10 19:01:35.855982 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:35.855955 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8b2f8b_d0fe_4263_865b_2a839e95789a.slice/crio-a61fe43d7f27c70872695e69d9ac98daf375a425e7ca292b749d5a964c3719d3 WatchSource:0}: Error finding container a61fe43d7f27c70872695e69d9ac98daf375a425e7ca292b749d5a964c3719d3: Status 404 returned error can't find the container with id a61fe43d7f27c70872695e69d9ac98daf375a425e7ca292b749d5a964c3719d3 Jul 10 19:01:35.863432 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:35.863407 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c87d7859f-vvbzf"] Jul 10 19:01:35.866158 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:35.866137 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f8db00_c888_4ce8_8637_bb43dc72920e.slice/crio-6fc6988b53848644cf8c172683b93694be1d9d865811abeb2b017e5c597d367c WatchSource:0}: Error finding container 6fc6988b53848644cf8c172683b93694be1d9d865811abeb2b017e5c597d367c: Status 404 returned error can't find the container with id 6fc6988b53848644cf8c172683b93694be1d9d865811abeb2b017e5c597d367c Jul 10 19:01:36.216159 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.215947 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qhkzc" event={"ID":"2f888fa4-b9d3-45dc-bc95-9801d7ecef40","Type":"ContainerStarted","Data":"d0a057b87308f4b774f71e84128d66334ed8891204e02cca29a6ad782fe30f5b"} Jul 10 19:01:36.218478 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.218425 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" event={"ID":"b50a5fd9-7250-458b-80c0-24b1fa7e35a3","Type":"ContainerStarted","Data":"642cd88e1c2f5d8c2000ea5517f70585db6937cc11c65cd3decb02bc3aeb2d0c"} Jul 10 19:01:36.220483 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.220423 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" event={"ID":"8f8b2f8b-d0fe-4263-865b-2a839e95789a","Type":"ContainerStarted","Data":"a61fe43d7f27c70872695e69d9ac98daf375a425e7ca292b749d5a964c3719d3"} Jul 10 19:01:36.222662 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.222604 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" event={"ID":"e85f93bd-0c5e-403f-b443-9cc24790e26c","Type":"ContainerStarted","Data":"b66e26387b803d858b8c10cff95ba9e47d6917112762e757b726d31831589cc0"} Jul 10 19:01:36.225142 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.225105 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" event={"ID":"83f8db00-c888-4ce8-8637-bb43dc72920e","Type":"ContainerStarted","Data":"66eefa562c9996231802d15e5b8c7ba43dea48b1c4b2641a533fd538b53e0482"} Jul 10 19:01:36.225276 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.225150 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:01:36.225276 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.225165 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" event={"ID":"83f8db00-c888-4ce8-8637-bb43dc72920e","Type":"ContainerStarted","Data":"6fc6988b53848644cf8c172683b93694be1d9d865811abeb2b017e5c597d367c"} Jul 10 19:01:36.228014 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.227990 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8b6wq" event={"ID":"ae7dfa1d-acb5-4192-91f8-241e98c91f61","Type":"ContainerStarted","Data":"a16771f6bf667218cc0e46c72a4fcf9992fa7afe58d68b7353c3a692b8a4fbc0"} Jul 10 19:01:36.232306 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.231700 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9rcmj" event={"ID":"7f8eb8de-8a4d-4081-a196-93d78a90535d","Type":"ContainerStarted","Data":"f63af219dfc07c3499fc0f1a06a14d07df36d96fadb596c47b6a64984184b031"} Jul 10 19:01:36.232306 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.232266 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:01:36.236693 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.235492 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qhkzc" podStartSLOduration=66.56437554 podStartE2EDuration="1m12.235474299s" totalImagesPullingTime="5.671098759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:36.234855412 +0000 UTC m=+72.985760312" watchObservedRunningTime="2026-07-10 19:01:36.235474299 +0000 UTC m=+72.986379175" Jul 10 19:01:36.241492 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.241474 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ksmpt" event={"ID":"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186","Type":"ContainerStarted","Data":"2d4aec42457802e13aa471d2d30d95d0fcc2676502a31b9b2f724d601a4c2417"} Jul 10 19:01:36.241597 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.241507 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ksmpt" event={"ID":"a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186","Type":"ContainerStarted","Data":"f54ea892f9a76c886ead902dcc5f5c178c8c71102ff696d64766a92d89cc4116"} Jul 10 19:01:36.241949 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.241928 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:36.244777 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.244756 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" event={"ID":"50137b74-0aab-4bc8-9568-06486fe76427","Type":"ContainerStarted","Data":"b2254e7fc0b88159585ea36a88c87998c38ac656f9d6f899e34d7d67c2092a8c"} Jul 10 19:01:36.259435 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.259246 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" podStartSLOduration=2.2592282089999998 podStartE2EDuration="2.259228209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:36.257712715 +0000 UTC m=+73.008617591" watchObservedRunningTime="2026-07-10 19:01:36.259228209 +0000 UTC m=+73.010133084" Jul 10 19:01:36.279089 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:36.278935 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9rcmj" podStartSLOduration=66.953938038 podStartE2EDuration="1m12.278914884s" totalImagesPullingTime="5.324976846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:36.277478509 +0000 UTC m=+73.028383403" watchObservedRunningTime="2026-07-10 19:01:36.278914884 +0000 UTC m=+73.029819763" Jul 10 19:01:38.264029 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:38.263911 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" event={"ID":"e85f93bd-0c5e-403f-b443-9cc24790e26c","Type":"ContainerStarted","Data":"c022da2219c59b82f43526e819b29d4e8a647ee0507f7912d85d7593dfdabce1"} Jul 10 19:01:38.287876 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:38.287817 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ksmpt" podStartSLOduration=35.476499198 podStartE2EDuration="39.287779851s" totalImagesPullingTime="3.811280653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:36.308461168 +0000 UTC m=+73.059366070" watchObservedRunningTime="2026-07-10 19:01:38.287779851 +0000 UTC m=+75.038684730" Jul 10 19:01:44.294820 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.294756 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" event={"ID":"b50a5fd9-7250-458b-80c0-24b1fa7e35a3","Type":"ContainerStarted","Data":"b17e2a5e3587e312f33e71828eb5f6c7ddb23afc7a966e951bb2c54d47768ee8"} Jul 10 19:01:44.296555 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.296531 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" event={"ID":"8f8b2f8b-d0fe-4263-865b-2a839e95789a","Type":"ContainerStarted","Data":"7d99d5d019682334c765aea201cdcc695074165370ed765149148fbb69967e3c"} Jul 10 19:01:44.296697 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.296683 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:44.298298 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.298267 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8b6wq" event={"ID":"ae7dfa1d-acb5-4192-91f8-241e98c91f61","Type":"ContainerStarted","Data":"0a7b8488062c985239a20f8bd42a41980cef33ae3d154f6ddcb7604fa8bc787b"} Jul 10 19:01:44.298806 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.298763 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" Jul 10 19:01:44.300397 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.300364 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" event={"ID":"50137b74-0aab-4bc8-9568-06486fe76427","Type":"ContainerStarted","Data":"387cb51424fe5d12756a15d75d4355e407f2e78e4f2be732e20393b02b9ee60d"} Jul 10 19:01:44.300526 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.300416 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" event={"ID":"50137b74-0aab-4bc8-9568-06486fe76427","Type":"ContainerStarted","Data":"761266ece963a43b1df98fdbf3092729985bd58072ac3e09e643ea82bd56c7de"} Jul 10 19:01:44.313253 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.313179 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-99qsw" podStartSLOduration=8.344826165 podStartE2EDuration="10.313163373s" totalImagesPullingTime="1.968337208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:38.288103966 +0000 UTC m=+75.039008842" watchObservedRunningTime="2026-07-10 19:01:44.313163373 +0000 UTC m=+81.064068250" Jul 10 19:01:44.313426 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.313285 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d6f6d66-tgwrf" podStartSLOduration=2.242194423 podStartE2EDuration="10.31327632s" totalImagesPullingTime="8.071081897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:44.312649851 +0000 UTC m=+81.063554725" watchObservedRunningTime="2026-07-10 19:01:44.31327632 +0000 UTC m=+81.064181197" Jul 10 19:01:44.333846 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.333775 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-mgrlx" podStartSLOduration=34.36591992 podStartE2EDuration="42.333758613s" totalImagesPullingTime="7.967838693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:44.332501188 +0000 UTC m=+81.083406062" watchObservedRunningTime="2026-07-10 19:01:44.333758613 +0000 UTC m=+81.084663464" Jul 10 19:01:44.357783 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:44.357725 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-565c6f4f58-qnk6c" podStartSLOduration=2.566152746 podStartE2EDuration="10.357706929s" totalImagesPullingTime="7.791554183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:44.357451763 +0000 UTC m=+81.108356637" watchObservedRunningTime="2026-07-10 19:01:44.357706929 +0000 UTC m=+81.108611803" Jul 10 19:01:45.226508 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.226385 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dqlrc"] Jul 10 19:01:45.229011 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.228988 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.232106 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.232002 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 19:01:45.232590 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.232571 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 19:01:45.233022 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.232996 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m7zfp\"" Jul 10 19:01:45.233148 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.233097 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 19:01:45.233246 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.233170 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 19:01:45.233246 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.233103 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 19:01:45.233351 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.233285 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 19:01:45.308359 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.308280 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8b6wq" event={"ID":"ae7dfa1d-acb5-4192-91f8-241e98c91f61","Type":"ContainerStarted","Data":"9aa0975e658802817978a6fccde79a62f6286fc4577a0e3ff4366e2a3dcafc4a"} Jul 10 19:01:45.340120 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340083 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-sys\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.340275 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340131 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4klr\" (UniqueName: \"kubernetes.io/projected/cd66431b-b049-486b-b194-31e528e42fda-kube-api-access-p4klr\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.340275 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340205 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.340365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340288 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-wtmp\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.340365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340343 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-tls\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.340484 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340376 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.340484 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340402 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-textfile\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.340484 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340451 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-root\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.340597 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.340499 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd66431b-b049-486b-b194-31e528e42fda-metrics-client-ca\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442046 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.441845 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd66431b-b049-486b-b194-31e528e42fda-metrics-client-ca\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442277 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-sys\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442317 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4klr\" (UniqueName: \"kubernetes.io/projected/cd66431b-b049-486b-b194-31e528e42fda-kube-api-access-p4klr\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442354 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442510 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442438 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-wtmp\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442510 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442478 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-tls\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442618 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442513 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442618 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442543 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-textfile\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442618 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442578 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-root\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442618 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442590 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd66431b-b049-486b-b194-31e528e42fda-metrics-client-ca\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442660 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-root\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.442908 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442869 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-wtmp\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.445195 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.445171 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-textfile\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.445437 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.445422 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.445600 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.442909 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd66431b-b049-486b-b194-31e528e42fda-sys\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.445756 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.445729 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-tls\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.446033 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.446010 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd66431b-b049-486b-b194-31e528e42fda-node-exporter-accelerators-collector-config\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.453054 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.453032 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4klr\" (UniqueName: \"kubernetes.io/projected/cd66431b-b049-486b-b194-31e528e42fda-kube-api-access-p4klr\") pod \"node-exporter-dqlrc\" (UID: \"cd66431b-b049-486b-b194-31e528e42fda\") " pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.540973 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:45.540888 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dqlrc" Jul 10 19:01:45.552788 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:45.552750 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd66431b_b049_486b_b194_31e528e42fda.slice/crio-185bd2a0631495e2611bfd96c9cbbd259b7b619229e261ededd6ec473979ec9b WatchSource:0}: Error finding container 185bd2a0631495e2611bfd96c9cbbd259b7b619229e261ededd6ec473979ec9b: Status 404 returned error can't find the container with id 185bd2a0631495e2611bfd96c9cbbd259b7b619229e261ededd6ec473979ec9b Jul 10 19:01:46.314389 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:46.314359 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqlrc" event={"ID":"cd66431b-b049-486b-b194-31e528e42fda","Type":"ContainerStarted","Data":"185bd2a0631495e2611bfd96c9cbbd259b7b619229e261ededd6ec473979ec9b"} Jul 10 19:01:47.267024 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:47.266992 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ksmpt" Jul 10 19:01:47.321972 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:47.321933 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8b6wq" event={"ID":"ae7dfa1d-acb5-4192-91f8-241e98c91f61","Type":"ContainerStarted","Data":"6ebe58061eb3f6e31af001e7717c4ef10e990282905662cdec4fa4c30e7d72a6"} Jul 10 19:01:47.350206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:47.350141 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8b6wq" podStartSLOduration=35.053884464 podStartE2EDuration="46.350117319s" totalImagesPullingTime="11.296232855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:01:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:47.348485014 +0000 UTC m=+84.099389914" watchObservedRunningTime="2026-07-10 19:01:47.350117319 +0000 UTC m=+84.101022193" Jul 10 19:01:48.327089 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:48.327056 2549 generic.go:356] "Generic (PLEG): container finished" podID="cd66431b-b049-486b-b194-31e528e42fda" containerID="5e8864ea7f5a2501aa120479cf63ab23ff9de1c20ef4fd5afeb87332c3aea494" exitCode=0 Jul 10 19:01:48.327465 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:48.327130 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqlrc" event={"ID":"cd66431b-b049-486b-b194-31e528e42fda","Type":"ContainerDied","Data":"5e8864ea7f5a2501aa120479cf63ab23ff9de1c20ef4fd5afeb87332c3aea494"} Jul 10 19:01:49.337143 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:49.337105 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqlrc" event={"ID":"cd66431b-b049-486b-b194-31e528e42fda","Type":"ContainerStarted","Data":"887c944b74f97894a6d7a169e0ca2a34ba1384b9b14ca5ba0cd9770726f37d2b"} Jul 10 19:01:49.337525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:49.337183 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dqlrc" event={"ID":"cd66431b-b049-486b-b194-31e528e42fda","Type":"ContainerStarted","Data":"2c8c931caa982a45d3ef8531f12fa60ba39fca064d40ca95a95c001e4be893c1"} Jul 10 19:01:49.366585 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:49.366528 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dqlrc" podStartSLOduration=2.352144787 podStartE2EDuration="4.366508492s" totalImagesPullingTime="1.943010381s" totalInitContainerRuntime="71.353324ms" isStatefulPod=true podCreationTimestamp="2026-07-10 19:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:49.365586575 +0000 UTC m=+86.116491474" watchObservedRunningTime="2026-07-10 19:01:49.366508492 +0000 UTC m=+86.117413367" Jul 10 19:01:49.961260 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:49.961221 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt"] Jul 10 19:01:49.963426 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:49.963410 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:49.965884 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:49.965859 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 19:01:49.965884 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:49.965877 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8xcx9\"" Jul 10 19:01:49.976680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:49.976655 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt"] Jul 10 19:01:50.082685 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.082654 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2db73e7c-4a11-4d2b-ac12-65ec881419f8-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-wzlwt\" (UID: \"2db73e7c-4a11-4d2b-ac12-65ec881419f8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:50.082855 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.082724 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kph7h\" (UniqueName: \"kubernetes.io/projected/2db73e7c-4a11-4d2b-ac12-65ec881419f8-kube-api-access-kph7h\") pod \"monitoring-plugin-5f68df7f58-wzlwt\" (UID: \"2db73e7c-4a11-4d2b-ac12-65ec881419f8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:50.184016 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.183983 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kph7h\" (UniqueName: \"kubernetes.io/projected/2db73e7c-4a11-4d2b-ac12-65ec881419f8-kube-api-access-kph7h\") pod \"monitoring-plugin-5f68df7f58-wzlwt\" (UID: \"2db73e7c-4a11-4d2b-ac12-65ec881419f8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:50.184183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.184046 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2db73e7c-4a11-4d2b-ac12-65ec881419f8-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-wzlwt\" (UID: \"2db73e7c-4a11-4d2b-ac12-65ec881419f8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:50.184227 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:50.184176 2549 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 10 19:01:50.184263 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:01:50.184254 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db73e7c-4a11-4d2b-ac12-65ec881419f8-monitoring-plugin-cert podName:2db73e7c-4a11-4d2b-ac12-65ec881419f8 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:50.684230196 +0000 UTC m=+87.435135066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/2db73e7c-4a11-4d2b-ac12-65ec881419f8-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-wzlwt" (UID: "2db73e7c-4a11-4d2b-ac12-65ec881419f8") : secret "monitoring-plugin-cert" not found Jul 10 19:01:50.193014 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.192986 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kph7h\" (UniqueName: \"kubernetes.io/projected/2db73e7c-4a11-4d2b-ac12-65ec881419f8-kube-api-access-kph7h\") pod \"monitoring-plugin-5f68df7f58-wzlwt\" (UID: \"2db73e7c-4a11-4d2b-ac12-65ec881419f8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:50.688462 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.688424 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2db73e7c-4a11-4d2b-ac12-65ec881419f8-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-wzlwt\" (UID: \"2db73e7c-4a11-4d2b-ac12-65ec881419f8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:50.691130 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.691108 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2db73e7c-4a11-4d2b-ac12-65ec881419f8-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-wzlwt\" (UID: \"2db73e7c-4a11-4d2b-ac12-65ec881419f8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:50.778653 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.778620 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7897957d6c-dkb68"] Jul 10 19:01:50.780849 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.780832 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.783880 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.783837 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 19:01:50.784024 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.783913 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 19:01:50.784024 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.783837 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 19:01:50.784024 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.783842 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 19:01:50.784755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.784741 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 19:01:50.784889 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.784771 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 19:01:50.784889 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.784827 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 19:01:50.784889 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.784833 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pmw2s\"" Jul 10 19:01:50.791784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.791766 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 19:01:50.795679 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.795647 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7897957d6c-dkb68"] Jul 10 19:01:50.873001 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.872967 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:50.890232 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.890197 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-trusted-ca-bundle\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.890363 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.890244 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-oauth-serving-cert\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.890363 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.890285 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc6tg\" (UniqueName: \"kubernetes.io/projected/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-kube-api-access-mc6tg\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.890363 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.890343 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-oauth-config\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.890499 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.890395 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-service-ca\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.890499 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.890431 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-serving-cert\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.890499 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.890458 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-config\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.991203 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.991173 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-trusted-ca-bundle\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.991381 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.991205 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-oauth-serving-cert\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.991381 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.991233 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc6tg\" (UniqueName: \"kubernetes.io/projected/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-kube-api-access-mc6tg\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.991381 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.991271 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-oauth-config\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.991381 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.991290 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-service-ca\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.991381 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.991311 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-serving-cert\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.991381 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.991327 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-config\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.992046 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.992022 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-oauth-serving-cert\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.992148 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.992095 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-service-ca\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.992148 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.992117 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-config\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.992219 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.992154 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-trusted-ca-bundle\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.994532 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.994511 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-serving-cert\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:50.994586 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:50.994565 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-oauth-config\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:51.004183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:51.004157 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc6tg\" (UniqueName: \"kubernetes.io/projected/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-kube-api-access-mc6tg\") pod \"console-7897957d6c-dkb68\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:51.013643 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:51.013627 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt"] Jul 10 19:01:51.016078 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:51.016057 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db73e7c_4a11_4d2b_ac12_65ec881419f8.slice/crio-e20e75e6254a17debbecef77c1922969788f292dbfa9b05864e9cc393fc9d074 WatchSource:0}: Error finding container e20e75e6254a17debbecef77c1922969788f292dbfa9b05864e9cc393fc9d074: Status 404 returned error can't find the container with id e20e75e6254a17debbecef77c1922969788f292dbfa9b05864e9cc393fc9d074 Jul 10 19:01:51.091005 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:51.090971 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:01:51.229074 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:51.228942 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7897957d6c-dkb68"] Jul 10 19:01:51.231694 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:01:51.231666 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78cebfc_7bbc_4731_92d1_a76bbfaa8a85.slice/crio-ec9978f492df29d07a3db8bf7c30066475a43207a1d68a5a539d6fb149be8ff8 WatchSource:0}: Error finding container ec9978f492df29d07a3db8bf7c30066475a43207a1d68a5a539d6fb149be8ff8: Status 404 returned error can't find the container with id ec9978f492df29d07a3db8bf7c30066475a43207a1d68a5a539d6fb149be8ff8 Jul 10 19:01:51.345518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:51.345487 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" event={"ID":"2db73e7c-4a11-4d2b-ac12-65ec881419f8","Type":"ContainerStarted","Data":"e20e75e6254a17debbecef77c1922969788f292dbfa9b05864e9cc393fc9d074"} Jul 10 19:01:51.346892 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:51.346865 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7897957d6c-dkb68" event={"ID":"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85","Type":"ContainerStarted","Data":"ec9978f492df29d07a3db8bf7c30066475a43207a1d68a5a539d6fb149be8ff8"} Jul 10 19:01:53.358358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:53.358312 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" event={"ID":"2db73e7c-4a11-4d2b-ac12-65ec881419f8","Type":"ContainerStarted","Data":"370ef58ae96fbb2009535c7242592fbbbbb939c995a082ef881e432e649f1ea9"} Jul 10 19:01:53.358761 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:53.358506 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:53.363672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:53.363644 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" Jul 10 19:01:53.380475 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:53.380419 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-wzlwt" podStartSLOduration=2.908749575 podStartE2EDuration="4.380402531s" totalImagesPullingTime="1.471652956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:53.379521528 +0000 UTC m=+90.130426404" watchObservedRunningTime="2026-07-10 19:01:53.380402531 +0000 UTC m=+90.131307407" Jul 10 19:01:55.368585 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:55.368546 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7897957d6c-dkb68" event={"ID":"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85","Type":"ContainerStarted","Data":"9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9"} Jul 10 19:01:55.389870 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:55.389814 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7897957d6c-dkb68" podStartSLOduration=2.011435956 podStartE2EDuration="5.389773966s" totalImagesPullingTime="3.37833801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:55.389088169 +0000 UTC m=+92.139993043" watchObservedRunningTime="2026-07-10 19:01:55.389773966 +0000 UTC m=+92.140678841" Jul 10 19:01:57.262008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:01:57.261978 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c87d7859f-vvbzf" Jul 10 19:02:01.091956 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:01.091911 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:02:01.092418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:01.092077 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:02:01.097228 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:01.097207 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:02:01.401280 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:01.401198 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:02:08.266914 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:08.266882 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9rcmj" Jul 10 19:02:14.746625 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:14.746589 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7897957d6c-dkb68"] Jul 10 19:02:36.529159 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:36.529126 2549 generic.go:356] "Generic (PLEG): container finished" podID="2c63a7aa-df11-4c44-bdc5-e19b075bf311" containerID="ac1afd3a7b7f6955dda4f3af3d852701fc029b81c536e682376c07736afea0ce" exitCode=0 Jul 10 19:02:36.529547 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:36.529197 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" event={"ID":"2c63a7aa-df11-4c44-bdc5-e19b075bf311","Type":"ContainerDied","Data":"ac1afd3a7b7f6955dda4f3af3d852701fc029b81c536e682376c07736afea0ce"} Jul 10 19:02:36.529612 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:36.529573 2549 scope.go:117] "RemoveContainer" containerID="ac1afd3a7b7f6955dda4f3af3d852701fc029b81c536e682376c07736afea0ce" Jul 10 19:02:36.530816 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:36.530767 2549 generic.go:356] "Generic (PLEG): container finished" podID="f20e4509-f5be-402b-bf6f-1178cbbff0f2" containerID="740a4f606c48a260229b0638d709f4b0cfa828da13e748e340004a1905519ce0" exitCode=0 Jul 10 19:02:36.530933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:36.530809 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" event={"ID":"f20e4509-f5be-402b-bf6f-1178cbbff0f2","Type":"ContainerDied","Data":"740a4f606c48a260229b0638d709f4b0cfa828da13e748e340004a1905519ce0"} Jul 10 19:02:36.531133 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:36.531118 2549 scope.go:117] "RemoveContainer" containerID="740a4f606c48a260229b0638d709f4b0cfa828da13e748e340004a1905519ce0" Jul 10 19:02:37.536299 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:37.536260 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-6lmf8" event={"ID":"f20e4509-f5be-402b-bf6f-1178cbbff0f2","Type":"ContainerStarted","Data":"249947008c04f5e1c77f8bac209615c632665a3a66165e8a02f6e2e25dbb4c9f"} Jul 10 19:02:37.537992 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:37.537970 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-7g2d7" event={"ID":"2c63a7aa-df11-4c44-bdc5-e19b075bf311","Type":"ContainerStarted","Data":"ea5c09fd3bf2031e75bae513c51f6a09da195fe4609e3dd9cbb9c19c99e66a17"} Jul 10 19:02:39.768923 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:39.768854 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7897957d6c-dkb68" podUID="e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" containerName="console" containerID="cri-o://9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9" gracePeriod=15 Jul 10 19:02:40.019188 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.019133 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:02:40.092209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092173 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-service-ca\") pod \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " Jul 10 19:02:40.092370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092216 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc6tg\" (UniqueName: \"kubernetes.io/projected/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-kube-api-access-mc6tg\") pod \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " Jul 10 19:02:40.092370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092240 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-oauth-serving-cert\") pod \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " Jul 10 19:02:40.092370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092271 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-trusted-ca-bundle\") pod \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " Jul 10 19:02:40.092370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092293 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-config\") pod \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " Jul 10 19:02:40.092370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092336 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-oauth-config\") pod \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " Jul 10 19:02:40.092370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092366 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-serving-cert\") pod \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\" (UID: \"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85\") " Jul 10 19:02:40.092758 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092732 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-service-ca" (OuterVolumeSpecName: "service-ca") pod "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" (UID: "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:02:40.092853 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092753 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" (UID: "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:02:40.092853 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092760 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" (UID: "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:02:40.092853 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.092770 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-config" (OuterVolumeSpecName: "console-config") pod "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" (UID: "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:02:40.094656 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.094629 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" (UID: "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:02:40.094734 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.094656 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-kube-api-access-mc6tg" (OuterVolumeSpecName: "kube-api-access-mc6tg") pod "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" (UID: "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85"). InnerVolumeSpecName "kube-api-access-mc6tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:02:40.094770 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.094729 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" (UID: "e78cebfc-7bbc-4731-92d1-a76bbfaa8a85"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:02:40.193435 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.193404 2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:02:40.193435 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.193433 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc6tg\" (UniqueName: \"kubernetes.io/projected/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-kube-api-access-mc6tg\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:02:40.193435 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.193444 2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-oauth-serving-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:02:40.193635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.193452 2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-trusted-ca-bundle\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:02:40.193635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.193461 2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-config\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:02:40.193635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.193469 2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-oauth-config\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:02:40.193635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.193477 2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85-console-serving-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:02:40.549894 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.549862 2549 generic.go:356] "Generic (PLEG): container finished" podID="e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" containerID="9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9" exitCode=0 Jul 10 19:02:40.550051 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.549911 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7897957d6c-dkb68" Jul 10 19:02:40.550051 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.549951 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7897957d6c-dkb68" event={"ID":"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85","Type":"ContainerDied","Data":"9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9"} Jul 10 19:02:40.550051 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.550015 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7897957d6c-dkb68" event={"ID":"e78cebfc-7bbc-4731-92d1-a76bbfaa8a85","Type":"ContainerDied","Data":"ec9978f492df29d07a3db8bf7c30066475a43207a1d68a5a539d6fb149be8ff8"} Jul 10 19:02:40.550051 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.550040 2549 scope.go:117] "RemoveContainer" containerID="9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9" Jul 10 19:02:40.558718 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.558698 2549 scope.go:117] "RemoveContainer" containerID="9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9" Jul 10 19:02:40.558999 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:02:40.558978 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9\": container with ID starting with 9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9 not found: ID does not exist" containerID="9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9" Jul 10 19:02:40.559064 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.559008 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9"} err="failed to get container status \"9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9\": rpc error: code = NotFound desc = could not find container \"9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9\": container with ID starting with 9bcb814f5ba90cc01f5daba5a75306d23e6c980c07d1470edd382b7e1b8d03f9 not found: ID does not exist" Jul 10 19:02:40.572306 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.572285 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7897957d6c-dkb68"] Jul 10 19:02:40.576362 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:40.576342 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7897957d6c-dkb68"] Jul 10 19:02:41.803839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:02:41.803787 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" path="/var/lib/kubelet/pods/e78cebfc-7bbc-4731-92d1-a76bbfaa8a85/volumes" Jul 10 19:03:02.420302 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.420263 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c776fb86f-z7zp7"] Jul 10 19:03:02.420942 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.420654 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" containerName="console" Jul 10 19:03:02.420942 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.420671 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" containerName="console" Jul 10 19:03:02.420942 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.420724 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78cebfc-7bbc-4731-92d1-a76bbfaa8a85" containerName="console" Jul 10 19:03:02.423949 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.423928 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.426290 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.426269 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pmw2s\"" Jul 10 19:03:02.426418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.426285 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 19:03:02.427231 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.427211 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 19:03:02.427384 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.427342 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 19:03:02.427465 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.427352 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 19:03:02.427465 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.427434 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 19:03:02.427663 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.427643 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 19:03:02.427785 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.427283 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 19:03:02.436365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.436341 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c776fb86f-z7zp7"] Jul 10 19:03:02.438336 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.438306 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 19:03:02.564418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.564382 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-console-config\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.564418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.564429 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-service-ca\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.564639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.564456 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-serving-cert\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.564639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.564531 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-oauth-config\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.564639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.564580 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-trusted-ca-bundle\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.564639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.564605 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4rx6\" (UniqueName: \"kubernetes.io/projected/18347256-714d-40ff-ab18-f2b259151f70-kube-api-access-c4rx6\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.564639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.564636 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-oauth-serving-cert\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.665493 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.665446 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-service-ca\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.665493 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.665494 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-serving-cert\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.665769 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.665522 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-oauth-config\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.665769 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.665551 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-trusted-ca-bundle\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.665769 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.665582 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4rx6\" (UniqueName: \"kubernetes.io/projected/18347256-714d-40ff-ab18-f2b259151f70-kube-api-access-c4rx6\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.665769 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.665614 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-oauth-serving-cert\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.665769 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.665663 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-console-config\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.666361 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.666327 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-service-ca\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.666361 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.666341 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-oauth-serving-cert\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.666686 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.666380 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-console-config\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.667084 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.667055 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-trusted-ca-bundle\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.668383 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.668364 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-oauth-config\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.668556 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.668537 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-serving-cert\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.674875 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.674780 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4rx6\" (UniqueName: \"kubernetes.io/projected/18347256-714d-40ff-ab18-f2b259151f70-kube-api-access-c4rx6\") pod \"console-7c776fb86f-z7zp7\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.735879 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.735841 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:02.878933 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:02.878908 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c776fb86f-z7zp7"] Jul 10 19:03:02.881562 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:03:02.881532 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18347256_714d_40ff_ab18_f2b259151f70.slice/crio-0dc3c5bd7ddabbd7f124e559e851937a94fe82a34b5fcee2197e6646181b7bad WatchSource:0}: Error finding container 0dc3c5bd7ddabbd7f124e559e851937a94fe82a34b5fcee2197e6646181b7bad: Status 404 returned error can't find the container with id 0dc3c5bd7ddabbd7f124e559e851937a94fe82a34b5fcee2197e6646181b7bad Jul 10 19:03:03.636547 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:03.636506 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c776fb86f-z7zp7" event={"ID":"18347256-714d-40ff-ab18-f2b259151f70","Type":"ContainerStarted","Data":"ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01"} Jul 10 19:03:03.636942 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:03.636558 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c776fb86f-z7zp7" event={"ID":"18347256-714d-40ff-ab18-f2b259151f70","Type":"ContainerStarted","Data":"0dc3c5bd7ddabbd7f124e559e851937a94fe82a34b5fcee2197e6646181b7bad"} Jul 10 19:03:03.655726 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:03.655680 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c776fb86f-z7zp7" podStartSLOduration=1.6556639880000001 podStartE2EDuration="1.655663988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:03:03.65428123 +0000 UTC m=+160.405186116" watchObservedRunningTime="2026-07-10 19:03:03.655663988 +0000 UTC m=+160.406568861" Jul 10 19:03:12.736448 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:12.736405 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:12.736880 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:12.736474 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:12.741502 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:12.741475 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:03:13.677414 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:03:13.677383 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:04:19.612037 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.612000 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk"] Jul 10 19:04:19.615012 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.614994 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" Jul 10 19:04:19.617787 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.617762 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 19:04:19.617892 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.617762 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 19:04:19.618652 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.618630 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-sx8n8\"" Jul 10 19:04:19.628724 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.628699 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk"] Jul 10 19:04:19.682285 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.682242 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f8851f8-b99c-4940-aed5-df19a0526acd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9jgvk\" (UID: \"7f8851f8-b99c-4940-aed5-df19a0526acd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" Jul 10 19:04:19.682285 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.682291 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cz89\" (UniqueName: \"kubernetes.io/projected/7f8851f8-b99c-4940-aed5-df19a0526acd-kube-api-access-4cz89\") pod \"cert-manager-cainjector-559cdd8c87-9jgvk\" (UID: \"7f8851f8-b99c-4940-aed5-df19a0526acd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" Jul 10 19:04:19.783629 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.783598 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f8851f8-b99c-4940-aed5-df19a0526acd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9jgvk\" (UID: \"7f8851f8-b99c-4940-aed5-df19a0526acd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" Jul 10 19:04:19.783629 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.783634 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cz89\" (UniqueName: \"kubernetes.io/projected/7f8851f8-b99c-4940-aed5-df19a0526acd-kube-api-access-4cz89\") pod \"cert-manager-cainjector-559cdd8c87-9jgvk\" (UID: \"7f8851f8-b99c-4940-aed5-df19a0526acd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" Jul 10 19:04:19.793182 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.793149 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7f8851f8-b99c-4940-aed5-df19a0526acd-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9jgvk\" (UID: \"7f8851f8-b99c-4940-aed5-df19a0526acd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" Jul 10 19:04:19.793307 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.793237 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cz89\" (UniqueName: \"kubernetes.io/projected/7f8851f8-b99c-4940-aed5-df19a0526acd-kube-api-access-4cz89\") pod \"cert-manager-cainjector-559cdd8c87-9jgvk\" (UID: \"7f8851f8-b99c-4940-aed5-df19a0526acd\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" Jul 10 19:04:19.923885 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:19.923780 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" Jul 10 19:04:20.060385 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:20.060363 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk"] Jul 10 19:04:20.063123 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:04:20.063079 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f8851f8_b99c_4940_aed5_df19a0526acd.slice/crio-3ceb9aa39145ef9a43300f21747ec3ec7da1bc9506cbbfc17b5042c4584c51cc WatchSource:0}: Error finding container 3ceb9aa39145ef9a43300f21747ec3ec7da1bc9506cbbfc17b5042c4584c51cc: Status 404 returned error can't find the container with id 3ceb9aa39145ef9a43300f21747ec3ec7da1bc9506cbbfc17b5042c4584c51cc Jul 10 19:04:20.926369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:20.926326 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" event={"ID":"7f8851f8-b99c-4940-aed5-df19a0526acd","Type":"ContainerStarted","Data":"3ceb9aa39145ef9a43300f21747ec3ec7da1bc9506cbbfc17b5042c4584c51cc"} Jul 10 19:04:23.941611 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:23.941572 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" event={"ID":"7f8851f8-b99c-4940-aed5-df19a0526acd","Type":"ContainerStarted","Data":"6029b70b7079c47618bb712228bcaa0ff397f80a60450d2253a5a9b5022691b3"} Jul 10 19:04:23.959323 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:23.959271 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9jgvk" podStartSLOduration=2.026651049 podStartE2EDuration="4.959255171s" totalImagesPullingTime="2.932604122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:23.958176772 +0000 UTC m=+240.709081655" watchObservedRunningTime="2026-07-10 19:04:23.959255171 +0000 UTC m=+240.710160045" Jul 10 19:04:51.575564 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.575485 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9"] Jul 10 19:04:51.578572 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.578556 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.582572 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.582549 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 19:04:51.582788 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.582768 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 19:04:51.583067 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.583052 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 19:04:51.583171 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.583151 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lhpvl\"" Jul 10 19:04:51.583356 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.583343 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 19:04:51.600515 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.600485 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9"] Jul 10 19:04:51.639016 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.638975 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f57f5f88-b93a-4886-8ad8-454598cbea0b-tmp\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.639179 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.639035 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vg2\" (UniqueName: \"kubernetes.io/projected/f57f5f88-b93a-4886-8ad8-454598cbea0b-kube-api-access-m8vg2\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.639179 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.639081 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f57f5f88-b93a-4886-8ad8-454598cbea0b-manifests\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.639179 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.639151 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f57f5f88-b93a-4886-8ad8-454598cbea0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.639325 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.639189 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f57f5f88-b93a-4886-8ad8-454598cbea0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.739680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.739643 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f57f5f88-b93a-4886-8ad8-454598cbea0b-tmp\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.739680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.739680 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vg2\" (UniqueName: \"kubernetes.io/projected/f57f5f88-b93a-4886-8ad8-454598cbea0b-kube-api-access-m8vg2\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.739952 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.739708 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f57f5f88-b93a-4886-8ad8-454598cbea0b-manifests\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.739952 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.739748 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f57f5f88-b93a-4886-8ad8-454598cbea0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.739952 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.739767 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f57f5f88-b93a-4886-8ad8-454598cbea0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.740219 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.740194 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f57f5f88-b93a-4886-8ad8-454598cbea0b-manifests\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.742366 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.742344 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f57f5f88-b93a-4886-8ad8-454598cbea0b-tmp\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.742506 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.742483 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f57f5f88-b93a-4886-8ad8-454598cbea0b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.742566 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.742506 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f57f5f88-b93a-4886-8ad8-454598cbea0b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.751927 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.751898 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vg2\" (UniqueName: \"kubernetes.io/projected/f57f5f88-b93a-4886-8ad8-454598cbea0b-kube-api-access-m8vg2\") pod \"opendatahub-operator-controller-manager-7668cd484b-hzpr9\" (UID: \"f57f5f88-b93a-4886-8ad8-454598cbea0b\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:51.889020 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:51.888937 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:52.037408 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:52.037380 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9"] Jul 10 19:04:52.039268 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:04:52.039244 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57f5f88_b93a_4886_8ad8_454598cbea0b.slice/crio-f279c070e13d5fef7a42e42653f4745bd1b0cd8b46daab55afb66033db79a3bd WatchSource:0}: Error finding container f279c070e13d5fef7a42e42653f4745bd1b0cd8b46daab55afb66033db79a3bd: Status 404 returned error can't find the container with id f279c070e13d5fef7a42e42653f4745bd1b0cd8b46daab55afb66033db79a3bd Jul 10 19:04:52.054386 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:52.054359 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" event={"ID":"f57f5f88-b93a-4886-8ad8-454598cbea0b","Type":"ContainerStarted","Data":"f279c070e13d5fef7a42e42653f4745bd1b0cd8b46daab55afb66033db79a3bd"} Jul 10 19:04:56.074326 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:56.074287 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" event={"ID":"f57f5f88-b93a-4886-8ad8-454598cbea0b","Type":"ContainerStarted","Data":"6aa4721e9e5e336324499d5e6652acc1136ebb10bc2040c2f3afa9cdb949f85a"} Jul 10 19:04:57.080132 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:57.080092 2549 generic.go:356] "Generic (PLEG): container finished" podID="f57f5f88-b93a-4886-8ad8-454598cbea0b" containerID="6aa4721e9e5e336324499d5e6652acc1136ebb10bc2040c2f3afa9cdb949f85a" exitCode=0 Jul 10 19:04:57.080758 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:57.080146 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" event={"ID":"f57f5f88-b93a-4886-8ad8-454598cbea0b","Type":"ContainerDied","Data":"6aa4721e9e5e336324499d5e6652acc1136ebb10bc2040c2f3afa9cdb949f85a"} Jul 10 19:04:58.086273 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:58.086238 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" event={"ID":"f57f5f88-b93a-4886-8ad8-454598cbea0b","Type":"ContainerStarted","Data":"6fd3dc8ad38de47141fe6d1899b19afb6a81e7b2ea4e90755a0f65d217e65650"} Jul 10 19:04:58.086644 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:58.086362 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:04:58.109672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:04:58.109622 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" podStartSLOduration=2.3352593649999998 podStartE2EDuration="7.109607926s" totalImagesPullingTime="3.680824205s" totalInitContainerRuntime="1.093524356s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:58.108701124 +0000 UTC m=+274.859606025" watchObservedRunningTime="2026-07-10 19:04:58.109607926 +0000 UTC m=+274.860512800" Jul 10 19:05:09.092342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:09.092312 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-hzpr9" Jul 10 19:05:11.511241 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.511204 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kj9l7"] Jul 10 19:05:11.520339 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.520317 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:11.523010 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.522986 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 19:05:11.523293 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.523277 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hqxd2\"" Jul 10 19:05:11.538991 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.538953 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kj9l7"] Jul 10 19:05:11.602464 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.602435 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49146843-8c17-438b-b5fd-231d5c18f7f3-cert\") pod \"odh-model-controller-7998c67dc4-kj9l7\" (UID: \"49146843-8c17-438b-b5fd-231d5c18f7f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:11.602464 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.602472 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svfnq\" (UniqueName: \"kubernetes.io/projected/49146843-8c17-438b-b5fd-231d5c18f7f3-kube-api-access-svfnq\") pod \"odh-model-controller-7998c67dc4-kj9l7\" (UID: \"49146843-8c17-438b-b5fd-231d5c18f7f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:11.703543 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.703498 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49146843-8c17-438b-b5fd-231d5c18f7f3-cert\") pod \"odh-model-controller-7998c67dc4-kj9l7\" (UID: \"49146843-8c17-438b-b5fd-231d5c18f7f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:11.703741 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.703555 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfnq\" (UniqueName: \"kubernetes.io/projected/49146843-8c17-438b-b5fd-231d5c18f7f3-kube-api-access-svfnq\") pod \"odh-model-controller-7998c67dc4-kj9l7\" (UID: \"49146843-8c17-438b-b5fd-231d5c18f7f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:11.703741 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:05:11.703651 2549 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 19:05:11.703741 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:05:11.703734 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49146843-8c17-438b-b5fd-231d5c18f7f3-cert podName:49146843-8c17-438b-b5fd-231d5c18f7f3 nodeName:}" failed. No retries permitted until 2026-07-10 19:05:12.203715986 +0000 UTC m=+288.954620841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/49146843-8c17-438b-b5fd-231d5c18f7f3-cert") pod "odh-model-controller-7998c67dc4-kj9l7" (UID: "49146843-8c17-438b-b5fd-231d5c18f7f3") : secret "odh-model-controller-webhook-cert" not found Jul 10 19:05:11.714641 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:11.714617 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfnq\" (UniqueName: \"kubernetes.io/projected/49146843-8c17-438b-b5fd-231d5c18f7f3-kube-api-access-svfnq\") pod \"odh-model-controller-7998c67dc4-kj9l7\" (UID: \"49146843-8c17-438b-b5fd-231d5c18f7f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:12.207624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.207590 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49146843-8c17-438b-b5fd-231d5c18f7f3-cert\") pod \"odh-model-controller-7998c67dc4-kj9l7\" (UID: \"49146843-8c17-438b-b5fd-231d5c18f7f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:12.210311 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.210288 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49146843-8c17-438b-b5fd-231d5c18f7f3-cert\") pod \"odh-model-controller-7998c67dc4-kj9l7\" (UID: \"49146843-8c17-438b-b5fd-231d5c18f7f3\") " pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:12.296226 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.296187 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bcngx"] Jul 10 19:05:12.298414 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.298398 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:12.300917 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.300898 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 19:05:12.301002 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.300903 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-m8jjq\"" Jul 10 19:05:12.308364 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.308335 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bcngx"] Jul 10 19:05:12.409742 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.409704 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405d5e66-7968-48e0-a384-b7cd60fd4db6-tls-certs\") pod \"model-serving-api-54d89bd5fc-bcngx\" (UID: \"405d5e66-7968-48e0-a384-b7cd60fd4db6\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:12.409948 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.409857 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplft\" (UniqueName: \"kubernetes.io/projected/405d5e66-7968-48e0-a384-b7cd60fd4db6-kube-api-access-mplft\") pod \"model-serving-api-54d89bd5fc-bcngx\" (UID: \"405d5e66-7968-48e0-a384-b7cd60fd4db6\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:12.430628 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.430603 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:12.511157 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.511079 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405d5e66-7968-48e0-a384-b7cd60fd4db6-tls-certs\") pod \"model-serving-api-54d89bd5fc-bcngx\" (UID: \"405d5e66-7968-48e0-a384-b7cd60fd4db6\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:12.511360 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.511208 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplft\" (UniqueName: \"kubernetes.io/projected/405d5e66-7968-48e0-a384-b7cd60fd4db6-kube-api-access-mplft\") pod \"model-serving-api-54d89bd5fc-bcngx\" (UID: \"405d5e66-7968-48e0-a384-b7cd60fd4db6\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:12.514274 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.514245 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405d5e66-7968-48e0-a384-b7cd60fd4db6-tls-certs\") pod \"model-serving-api-54d89bd5fc-bcngx\" (UID: \"405d5e66-7968-48e0-a384-b7cd60fd4db6\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:12.521200 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.521169 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplft\" (UniqueName: \"kubernetes.io/projected/405d5e66-7968-48e0-a384-b7cd60fd4db6-kube-api-access-mplft\") pod \"model-serving-api-54d89bd5fc-bcngx\" (UID: \"405d5e66-7968-48e0-a384-b7cd60fd4db6\") " pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:12.565589 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.565565 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kj9l7"] Jul 10 19:05:12.568401 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:05:12.568377 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49146843_8c17_438b_b5fd_231d5c18f7f3.slice/crio-3e7936eeec604a2bb9eb51cb6566ee68d4b717d701bef4759f1949e7162f1330 WatchSource:0}: Error finding container 3e7936eeec604a2bb9eb51cb6566ee68d4b717d701bef4759f1949e7162f1330: Status 404 returned error can't find the container with id 3e7936eeec604a2bb9eb51cb6566ee68d4b717d701bef4759f1949e7162f1330 Jul 10 19:05:12.609724 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.609683 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:12.753840 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:12.753741 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bcngx"] Jul 10 19:05:12.757152 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:05:12.757120 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405d5e66_7968_48e0_a384_b7cd60fd4db6.slice/crio-b082e35b7f3dd1f3916ee0d0e8855e8d5bc0ca73bcb623a9518edfe94d1c495d WatchSource:0}: Error finding container b082e35b7f3dd1f3916ee0d0e8855e8d5bc0ca73bcb623a9518edfe94d1c495d: Status 404 returned error can't find the container with id b082e35b7f3dd1f3916ee0d0e8855e8d5bc0ca73bcb623a9518edfe94d1c495d Jul 10 19:05:13.145414 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:13.145380 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" event={"ID":"49146843-8c17-438b-b5fd-231d5c18f7f3","Type":"ContainerStarted","Data":"3e7936eeec604a2bb9eb51cb6566ee68d4b717d701bef4759f1949e7162f1330"} Jul 10 19:05:13.146539 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:13.146509 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" event={"ID":"405d5e66-7968-48e0-a384-b7cd60fd4db6","Type":"ContainerStarted","Data":"b082e35b7f3dd1f3916ee0d0e8855e8d5bc0ca73bcb623a9518edfe94d1c495d"} Jul 10 19:05:16.161926 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.161896 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" event={"ID":"49146843-8c17-438b-b5fd-231d5c18f7f3","Type":"ContainerStarted","Data":"15c7d9da587fef7436705c19f042083ca123894b58a8f4497a45048d5bda4a2f"} Jul 10 19:05:16.162292 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.162025 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:16.163494 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.163463 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" event={"ID":"405d5e66-7968-48e0-a384-b7cd60fd4db6","Type":"ContainerStarted","Data":"90796bb92c9c6476880a4ec3029814f235e67c65e49c8e79fe4778d004fea2bf"} Jul 10 19:05:16.163607 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.163515 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:16.191965 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.191633 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" podStartSLOduration=1.662262753 podStartE2EDuration="5.191610149s" totalImagesPullingTime="3.529347396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:16.191497216 +0000 UTC m=+292.942402110" watchObservedRunningTime="2026-07-10 19:05:16.191610149 +0000 UTC m=+292.942515024" Jul 10 19:05:16.863690 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.863638 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" podStartSLOduration=2.335137474 podStartE2EDuration="4.863618252s" totalImagesPullingTime="2.528480778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:16.220222466 +0000 UTC m=+292.971127363" watchObservedRunningTime="2026-07-10 19:05:16.863618252 +0000 UTC m=+293.614523127" Jul 10 19:05:16.864707 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.864690 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv"] Jul 10 19:05:16.866995 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.866978 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:16.872695 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.872667 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 19:05:16.873862 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.873838 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 19:05:16.873988 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.873838 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wb8gg\"" Jul 10 19:05:16.874270 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.874254 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 19:05:16.874538 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.874526 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 19:05:16.883639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:16.883615 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv"] Jul 10 19:05:17.049311 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.049276 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54995f91-4f93-4673-a906-091f7cd21698-tls-certs\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.049470 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.049323 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54995f91-4f93-4673-a906-091f7cd21698-tmp\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.049470 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.049390 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tcf\" (UniqueName: \"kubernetes.io/projected/54995f91-4f93-4673-a906-091f7cd21698-kube-api-access-v2tcf\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.150339 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.150259 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54995f91-4f93-4673-a906-091f7cd21698-tls-certs\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.150339 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.150308 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54995f91-4f93-4673-a906-091f7cd21698-tmp\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.150339 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.150334 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tcf\" (UniqueName: \"kubernetes.io/projected/54995f91-4f93-4673-a906-091f7cd21698-kube-api-access-v2tcf\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.152617 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.152589 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54995f91-4f93-4673-a906-091f7cd21698-tmp\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.152954 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.152936 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54995f91-4f93-4673-a906-091f7cd21698-tls-certs\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.162133 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.162112 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tcf\" (UniqueName: \"kubernetes.io/projected/54995f91-4f93-4673-a906-091f7cd21698-kube-api-access-v2tcf\") pod \"kube-auth-proxy-58b4fc6b9d-c2ngv\" (UID: \"54995f91-4f93-4673-a906-091f7cd21698\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.176379 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.176353 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" Jul 10 19:05:17.362720 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:17.362699 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv"] Jul 10 19:05:17.365244 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:05:17.365217 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54995f91_4f93_4673_a906_091f7cd21698.slice/crio-dc6c61fc97a3cd3b8adf4b7266ccc93265e45864a6f340fdb1356219ff96dd55 WatchSource:0}: Error finding container dc6c61fc97a3cd3b8adf4b7266ccc93265e45864a6f340fdb1356219ff96dd55: Status 404 returned error can't find the container with id dc6c61fc97a3cd3b8adf4b7266ccc93265e45864a6f340fdb1356219ff96dd55 Jul 10 19:05:18.174096 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.174063 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" event={"ID":"54995f91-4f93-4673-a906-091f7cd21698","Type":"ContainerStarted","Data":"dc6c61fc97a3cd3b8adf4b7266ccc93265e45864a6f340fdb1356219ff96dd55"} Jul 10 19:05:18.763908 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.763867 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl"] Jul 10 19:05:18.765942 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.765925 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.770197 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.770177 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 19:05:18.770348 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.770177 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 19:05:18.770348 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.770178 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 19:05:18.770919 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.770901 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 19:05:18.770978 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.770965 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h8rzd\"" Jul 10 19:05:18.787717 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.787696 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl"] Jul 10 19:05:18.864786 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.864740 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.864786 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.864816 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.865045 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.864845 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2f8\" (UniqueName: \"kubernetes.io/projected/267d0dc5-ed6e-4977-889e-9c620744d07d-kube-api-access-7q2f8\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.865045 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.864884 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.865045 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.864956 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.865045 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.865013 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/267d0dc5-ed6e-4977-889e-9c620744d07d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.865179 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.865075 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.965977 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.965933 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.965977 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.965981 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.966272 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.966007 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/267d0dc5-ed6e-4977-889e-9c620744d07d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.966272 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.966044 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.966272 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.966083 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.966272 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.966107 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.966272 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.966128 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2f8\" (UniqueName: \"kubernetes.io/projected/267d0dc5-ed6e-4977-889e-9c620744d07d-kube-api-access-7q2f8\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.967519 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.967481 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.969014 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.968989 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.969102 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.969069 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.969396 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.969376 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/267d0dc5-ed6e-4977-889e-9c620744d07d-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.969699 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.969675 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/267d0dc5-ed6e-4977-889e-9c620744d07d-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.978151 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.978124 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267d0dc5-ed6e-4977-889e-9c620744d07d-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:18.978440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:18.978417 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2f8\" (UniqueName: \"kubernetes.io/projected/267d0dc5-ed6e-4977-889e-9c620744d07d-kube-api-access-7q2f8\") pod \"istiod-openshift-gateway-94bb8fbfd-nfgjl\" (UID: \"267d0dc5-ed6e-4977-889e-9c620744d07d\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:19.076964 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:19.076934 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:19.248300 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:19.248262 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl"] Jul 10 19:05:19.252228 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:05:19.252195 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267d0dc5_ed6e_4977_889e_9c620744d07d.slice/crio-db970f8e809a31376e4c090abdaa17b8310114b3d1f4827a1b8bee94883c46b4 WatchSource:0}: Error finding container db970f8e809a31376e4c090abdaa17b8310114b3d1f4827a1b8bee94883c46b4: Status 404 returned error can't find the container with id db970f8e809a31376e4c090abdaa17b8310114b3d1f4827a1b8bee94883c46b4 Jul 10 19:05:20.184807 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:20.184744 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" event={"ID":"267d0dc5-ed6e-4977-889e-9c620744d07d","Type":"ContainerStarted","Data":"db970f8e809a31376e4c090abdaa17b8310114b3d1f4827a1b8bee94883c46b4"} Jul 10 19:05:21.195440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:21.190705 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" event={"ID":"54995f91-4f93-4673-a906-091f7cd21698","Type":"ContainerStarted","Data":"14e5516840729ae0b22c1a115f6c2549882ffea7bdc74ad8d47442997a13c4e7"} Jul 10 19:05:21.222093 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:21.222030 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-c2ngv" podStartSLOduration=2.18679119 podStartE2EDuration="5.222013778s" totalImagesPullingTime="3.035222588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:21.217618553 +0000 UTC m=+297.968523454" watchObservedRunningTime="2026-07-10 19:05:21.222013778 +0000 UTC m=+297.972918660" Jul 10 19:05:23.678405 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:23.678370 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:05:23.678960 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:23.678608 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:05:23.688835 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:23.688810 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:05:23.689005 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:23.688864 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:05:23.695058 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:23.695038 2549 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 19:05:27.171899 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:27.171871 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kj9l7" Jul 10 19:05:27.175151 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:27.175126 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bcngx" Jul 10 19:05:37.296855 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:37.296782 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:05:37.297157 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:37.296884 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:05:38.266514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:38.266471 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" event={"ID":"267d0dc5-ed6e-4977-889e-9c620744d07d","Type":"ContainerStarted","Data":"1f59b43120a49f15c840e40f9dc062e512bafd8802d3782681d2bae476f3e5ef"} Jul 10 19:05:38.266720 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:38.266635 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:38.268684 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:38.268650 2549 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-nfgjl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:05:38.268838 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:38.268714 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" podUID="267d0dc5-ed6e-4977-889e-9c620744d07d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:05:39.273515 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.273478 2549 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-nfgjl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:05:39.273945 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.273553 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" podUID="267d0dc5-ed6e-4977-889e-9c620744d07d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:05:39.589916 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.589849 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" podStartSLOduration=3.54768041 podStartE2EDuration="21.589824911s" totalImagesPullingTime="18.042144501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:38.30175092 +0000 UTC m=+315.052655822" watchObservedRunningTime="2026-07-10 19:05:39.589824911 +0000 UTC m=+316.340729786" Jul 10 19:05:39.591700 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.591667 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg"] Jul 10 19:05:39.593901 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.593886 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.597518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.597496 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-csnjt\"" Jul 10 19:05:39.610617 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.610558 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg"] Jul 10 19:05:39.622114 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622083 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.622215 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622120 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.622215 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622155 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.622215 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622196 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvx27\" (UniqueName: \"kubernetes.io/projected/f4238c24-1222-4d9c-bc57-23759dbaf816-kube-api-access-cvx27\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.622314 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622237 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.622314 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622278 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.622391 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622310 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.622391 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622338 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.622391 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.622363 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4238c24-1222-4d9c-bc57-23759dbaf816-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723436 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723404 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723436 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723437 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723459 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723477 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4238c24-1222-4d9c-bc57-23759dbaf816-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723505 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723522 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723553 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723607 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvx27\" (UniqueName: \"kubernetes.io/projected/f4238c24-1222-4d9c-bc57-23759dbaf816-kube-api-access-cvx27\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.723680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723653 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.724055 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.723915 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.724116 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.724057 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.724171 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.724117 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.724340 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.724316 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4238c24-1222-4d9c-bc57-23759dbaf816-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.724445 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.724351 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.726427 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.726406 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.726577 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.726560 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.733527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.733506 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4238c24-1222-4d9c-bc57-23759dbaf816-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.736960 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.736936 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvx27\" (UniqueName: \"kubernetes.io/projected/f4238c24-1222-4d9c-bc57-23759dbaf816-kube-api-access-cvx27\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678rpldg\" (UID: \"f4238c24-1222-4d9c-bc57-23759dbaf816\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:39.904492 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:39.904401 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:40.040440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:40.040410 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg"] Jul 10 19:05:40.043314 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:05:40.043288 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4238c24_1222_4d9c_bc57_23759dbaf816.slice/crio-89d745325a305fdf1b773b8e1bd8cc3785edbefd869f21fdd02b54d61843e944 WatchSource:0}: Error finding container 89d745325a305fdf1b773b8e1bd8cc3785edbefd869f21fdd02b54d61843e944: Status 404 returned error can't find the container with id 89d745325a305fdf1b773b8e1bd8cc3785edbefd869f21fdd02b54d61843e944 Jul 10 19:05:40.045316 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:40.045300 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:05:40.278164 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:40.278076 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" event={"ID":"f4238c24-1222-4d9c-bc57-23759dbaf816","Type":"ContainerStarted","Data":"89d745325a305fdf1b773b8e1bd8cc3785edbefd869f21fdd02b54d61843e944"} Jul 10 19:05:42.274102 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:42.274071 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-nfgjl" Jul 10 19:05:42.450343 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:42.450306 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:05:42.450457 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:42.450378 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:05:42.450457 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:42.450405 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:05:43.294640 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:43.294606 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" event={"ID":"f4238c24-1222-4d9c-bc57-23759dbaf816","Type":"ContainerStarted","Data":"443eaef859fe8af4c8cb73e2123a6869b92825edbcbb9fbe853fb1e10ccd4b8c"} Jul 10 19:05:43.317911 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:43.317856 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" podStartSLOduration=1.91320504 podStartE2EDuration="4.317838427s" totalImagesPullingTime="2.404633387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:43.317100972 +0000 UTC m=+320.068005860" watchObservedRunningTime="2026-07-10 19:05:43.317838427 +0000 UTC m=+320.068743301" Jul 10 19:05:43.904935 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:43.904898 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:43.909500 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:43.909474 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:44.299714 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:44.299626 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:05:44.300610 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:05:44.300584 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678rpldg" Jul 10 19:06:47.844831 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.844774 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2"] Jul 10 19:06:47.849295 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.849266 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" Jul 10 19:06:47.853038 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.853008 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 19:06:47.853038 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.853019 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 19:06:47.853257 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.853075 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 19:06:47.854015 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.854001 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zf245\"" Jul 10 19:06:47.858828 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.858788 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2"] Jul 10 19:06:47.887745 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.887717 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-988qx\" (UniqueName: \"kubernetes.io/projected/4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2-kube-api-access-988qx\") pod \"dns-operator-controller-manager-864dd78c4c-v5nn2\" (UID: \"4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" Jul 10 19:06:47.988953 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:47.988919 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-988qx\" (UniqueName: \"kubernetes.io/projected/4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2-kube-api-access-988qx\") pod \"dns-operator-controller-manager-864dd78c4c-v5nn2\" (UID: \"4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" Jul 10 19:06:48.022171 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:48.022139 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-988qx\" (UniqueName: \"kubernetes.io/projected/4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2-kube-api-access-988qx\") pod \"dns-operator-controller-manager-864dd78c4c-v5nn2\" (UID: \"4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" Jul 10 19:06:48.160226 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:48.160135 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" Jul 10 19:06:48.302503 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:48.302463 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2"] Jul 10 19:06:48.304975 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:06:48.304954 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb2f3f0_2fb1_4c8c_b6c4_31e1e26eccb2.slice/crio-365bb1f955be816d1335dddc45fcc975d4d48bd336b0163a451edfc8782dce56 WatchSource:0}: Error finding container 365bb1f955be816d1335dddc45fcc975d4d48bd336b0163a451edfc8782dce56: Status 404 returned error can't find the container with id 365bb1f955be816d1335dddc45fcc975d4d48bd336b0163a451edfc8782dce56 Jul 10 19:06:48.577289 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:48.577254 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" event={"ID":"4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2","Type":"ContainerStarted","Data":"365bb1f955be816d1335dddc45fcc975d4d48bd336b0163a451edfc8782dce56"} Jul 10 19:06:50.545270 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:50.545234 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j"] Jul 10 19:06:50.548404 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:50.548378 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:06:50.550952 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:50.550923 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wwncx\"" Jul 10 19:06:50.566444 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:50.566423 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j"] Jul 10 19:06:50.609815 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:50.609759 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt79w\" (UniqueName: \"kubernetes.io/projected/2b83bc62-2c4e-48dd-ba71-61e193ff8e99-kube-api-access-tt79w\") pod \"limitador-operator-controller-manager-859d7b78bd-x948j\" (UID: \"2b83bc62-2c4e-48dd-ba71-61e193ff8e99\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:06:50.711290 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:50.711253 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt79w\" (UniqueName: \"kubernetes.io/projected/2b83bc62-2c4e-48dd-ba71-61e193ff8e99-kube-api-access-tt79w\") pod \"limitador-operator-controller-manager-859d7b78bd-x948j\" (UID: \"2b83bc62-2c4e-48dd-ba71-61e193ff8e99\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:06:50.723021 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:50.722996 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt79w\" (UniqueName: \"kubernetes.io/projected/2b83bc62-2c4e-48dd-ba71-61e193ff8e99-kube-api-access-tt79w\") pod \"limitador-operator-controller-manager-859d7b78bd-x948j\" (UID: \"2b83bc62-2c4e-48dd-ba71-61e193ff8e99\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:06:50.858031 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:50.857992 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:06:51.184936 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:51.184914 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j"] Jul 10 19:06:51.187421 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:06:51.187389 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b83bc62_2c4e_48dd_ba71_61e193ff8e99.slice/crio-a624cc54d20c901659d2b831f9274e971227a80675833236fba44ace9fc5039a WatchSource:0}: Error finding container a624cc54d20c901659d2b831f9274e971227a80675833236fba44ace9fc5039a: Status 404 returned error can't find the container with id a624cc54d20c901659d2b831f9274e971227a80675833236fba44ace9fc5039a Jul 10 19:06:51.596158 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:51.596122 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" event={"ID":"2b83bc62-2c4e-48dd-ba71-61e193ff8e99","Type":"ContainerStarted","Data":"a624cc54d20c901659d2b831f9274e971227a80675833236fba44ace9fc5039a"} Jul 10 19:06:51.597626 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:51.597599 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" event={"ID":"4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2","Type":"ContainerStarted","Data":"77d4cc86565c3ac2f2accdad8ebc1ff9d77354cfe5571f5db7b517a0619fa47c"} Jul 10 19:06:51.597753 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:51.597729 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" Jul 10 19:06:51.637340 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:51.637285 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" podStartSLOduration=1.845013112 podStartE2EDuration="4.637265003s" totalImagesPullingTime="2.792251891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:51.635703332 +0000 UTC m=+388.386608222" watchObservedRunningTime="2026-07-10 19:06:51.637265003 +0000 UTC m=+388.388169878" Jul 10 19:06:53.609374 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:53.609336 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" event={"ID":"2b83bc62-2c4e-48dd-ba71-61e193ff8e99","Type":"ContainerStarted","Data":"49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1"} Jul 10 19:06:53.609732 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:53.609420 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:06:53.643452 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:06:53.643403 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" podStartSLOduration=2.057228663 podStartE2EDuration="3.643379551s" totalImagesPullingTime="1.586150888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:06:53.641259132 +0000 UTC m=+390.392164030" watchObservedRunningTime="2026-07-10 19:06:53.643379551 +0000 UTC m=+390.394284425" Jul 10 19:07:02.604479 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:02.604448 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-v5nn2" Jul 10 19:07:04.616149 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:04.616121 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:07:07.429416 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.429384 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c655fb948-glxvt"] Jul 10 19:07:07.433210 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.433183 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.446229 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.446206 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c655fb948-glxvt"] Jul 10 19:07:07.555712 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.555679 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23f77890-1285-45ac-9195-a3502235a23d-console-oauth-config\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.555712 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.555716 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-service-ca\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.555943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.555737 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23f77890-1285-45ac-9195-a3502235a23d-console-serving-cert\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.555943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.555752 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-oauth-serving-cert\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.555943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.555829 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-console-config\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.555943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.555855 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6gw\" (UniqueName: \"kubernetes.io/projected/23f77890-1285-45ac-9195-a3502235a23d-kube-api-access-pm6gw\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.555943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.555922 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-trusted-ca-bundle\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.656873 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.656833 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23f77890-1285-45ac-9195-a3502235a23d-console-oauth-config\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.657069 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.656883 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-service-ca\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.657069 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.656916 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23f77890-1285-45ac-9195-a3502235a23d-console-serving-cert\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.657069 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.656936 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-oauth-serving-cert\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.657069 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.656960 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-console-config\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.657069 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.656979 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6gw\" (UniqueName: \"kubernetes.io/projected/23f77890-1285-45ac-9195-a3502235a23d-kube-api-access-pm6gw\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.657069 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.657064 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-trusted-ca-bundle\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.657817 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.657766 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-oauth-serving-cert\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.657964 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.657882 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-console-config\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.658033 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.657978 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-trusted-ca-bundle\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.658033 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.658021 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/23f77890-1285-45ac-9195-a3502235a23d-service-ca\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.659469 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.659450 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/23f77890-1285-45ac-9195-a3502235a23d-console-oauth-config\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.659587 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.659568 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/23f77890-1285-45ac-9195-a3502235a23d-console-serving-cert\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.666627 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.666603 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6gw\" (UniqueName: \"kubernetes.io/projected/23f77890-1285-45ac-9195-a3502235a23d-kube-api-access-pm6gw\") pod \"console-6c655fb948-glxvt\" (UID: \"23f77890-1285-45ac-9195-a3502235a23d\") " pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.742721 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.742648 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:07.880834 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:07.880775 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c655fb948-glxvt"] Jul 10 19:07:07.883063 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:07:07.883035 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f77890_1285_45ac_9195_a3502235a23d.slice/crio-c384d78fd4aba9be54d2d39879e5f5b2768348a3a827799dace7a715ab1d9c99 WatchSource:0}: Error finding container c384d78fd4aba9be54d2d39879e5f5b2768348a3a827799dace7a715ab1d9c99: Status 404 returned error can't find the container with id c384d78fd4aba9be54d2d39879e5f5b2768348a3a827799dace7a715ab1d9c99 Jul 10 19:07:08.680357 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:08.680320 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c655fb948-glxvt" event={"ID":"23f77890-1285-45ac-9195-a3502235a23d","Type":"ContainerStarted","Data":"c7149f524ce8902e14bd2264108dcc5f0c59e73711b6ff26c63e1fd773abe3db"} Jul 10 19:07:08.680715 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:08.680374 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c655fb948-glxvt" event={"ID":"23f77890-1285-45ac-9195-a3502235a23d","Type":"ContainerStarted","Data":"c384d78fd4aba9be54d2d39879e5f5b2768348a3a827799dace7a715ab1d9c99"} Jul 10 19:07:08.704034 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:08.703986 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c655fb948-glxvt" podStartSLOduration=1.7039699659999998 podStartE2EDuration="1.703969966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:08.702672624 +0000 UTC m=+405.453577498" watchObservedRunningTime="2026-07-10 19:07:08.703969966 +0000 UTC m=+405.454874839" Jul 10 19:07:10.849346 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:10.849310 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx"] Jul 10 19:07:10.852691 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:10.852674 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:10.855442 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:10.855421 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhzck\"" Jul 10 19:07:10.874186 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:10.874158 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx"] Jul 10 19:07:10.986691 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:10.986648 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2dk2\" (UniqueName: \"kubernetes.io/projected/67b95e08-9442-4613-9477-5c6cbda01ba5-kube-api-access-m2dk2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" (UID: \"67b95e08-9442-4613-9477-5c6cbda01ba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:10.986691 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:10.986694 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b95e08-9442-4613-9477-5c6cbda01ba5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" (UID: \"67b95e08-9442-4613-9477-5c6cbda01ba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:11.088000 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.087960 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2dk2\" (UniqueName: \"kubernetes.io/projected/67b95e08-9442-4613-9477-5c6cbda01ba5-kube-api-access-m2dk2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" (UID: \"67b95e08-9442-4613-9477-5c6cbda01ba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:11.088000 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.088003 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b95e08-9442-4613-9477-5c6cbda01ba5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" (UID: \"67b95e08-9442-4613-9477-5c6cbda01ba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:11.088408 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.088392 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b95e08-9442-4613-9477-5c6cbda01ba5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" (UID: \"67b95e08-9442-4613-9477-5c6cbda01ba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:11.098454 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.098427 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2dk2\" (UniqueName: \"kubernetes.io/projected/67b95e08-9442-4613-9477-5c6cbda01ba5-kube-api-access-m2dk2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" (UID: \"67b95e08-9442-4613-9477-5c6cbda01ba5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:11.162920 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.162835 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:11.305913 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.305876 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx"] Jul 10 19:07:11.309144 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:07:11.309115 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b95e08_9442_4613_9477_5c6cbda01ba5.slice/crio-b41d5cc6cd059691f1734e9e1ee42c3981c314d2959acde8a128c8a632f99c22 WatchSource:0}: Error finding container b41d5cc6cd059691f1734e9e1ee42c3981c314d2959acde8a128c8a632f99c22: Status 404 returned error can't find the container with id b41d5cc6cd059691f1734e9e1ee42c3981c314d2959acde8a128c8a632f99c22 Jul 10 19:07:11.598483 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.598449 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx"] Jul 10 19:07:11.603925 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.603894 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx"] Jul 10 19:07:11.637255 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.637198 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j"] Jul 10 19:07:11.637498 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.637459 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" podUID="2b83bc62-2c4e-48dd-ba71-61e193ff8e99" containerName="manager" containerID="cri-o://49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1" gracePeriod=2 Jul 10 19:07:11.644994 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.644924 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j"] Jul 10 19:07:11.646627 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.646602 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z"] Jul 10 19:07:11.647279 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.647256 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b83bc62-2c4e-48dd-ba71-61e193ff8e99" containerName="manager" Jul 10 19:07:11.647279 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.647280 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b83bc62-2c4e-48dd-ba71-61e193ff8e99" containerName="manager" Jul 10 19:07:11.647463 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.647370 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b83bc62-2c4e-48dd-ba71-61e193ff8e99" containerName="manager" Jul 10 19:07:11.650513 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.650493 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:11.654786 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:11.654737 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x948j\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="2b83bc62-2c4e-48dd-ba71-61e193ff8e99" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:07:11.668526 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.668494 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf"] Jul 10 19:07:11.671579 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.671561 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" Jul 10 19:07:11.683566 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.683541 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z"] Jul 10 19:07:11.689782 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.689757 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf"] Jul 10 19:07:11.692601 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.692571 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6lx2\" (UniqueName: \"kubernetes.io/projected/549e448e-4522-41f1-95e2-dffcb399a31c-kube-api-access-b6lx2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c9x5z\" (UID: \"549e448e-4522-41f1-95e2-dffcb399a31c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:11.692729 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.692646 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/549e448e-4522-41f1-95e2-dffcb399a31c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c9x5z\" (UID: \"549e448e-4522-41f1-95e2-dffcb399a31c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:11.703625 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:11.703583 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-x948j\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="2b83bc62-2c4e-48dd-ba71-61e193ff8e99" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:07:11.793722 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.793682 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6lx2\" (UniqueName: \"kubernetes.io/projected/549e448e-4522-41f1-95e2-dffcb399a31c-kube-api-access-b6lx2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c9x5z\" (UID: \"549e448e-4522-41f1-95e2-dffcb399a31c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:11.793911 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.793735 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/549e448e-4522-41f1-95e2-dffcb399a31c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c9x5z\" (UID: \"549e448e-4522-41f1-95e2-dffcb399a31c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:11.793911 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.793810 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5hlq\" (UniqueName: \"kubernetes.io/projected/79cbea14-e7aa-448a-9e87-cb4f9005dc31-kube-api-access-w5hlq\") pod \"limitador-operator-controller-manager-859d7b78bd-6mhsf\" (UID: \"79cbea14-e7aa-448a-9e87-cb4f9005dc31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" Jul 10 19:07:11.794173 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.794152 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/549e448e-4522-41f1-95e2-dffcb399a31c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c9x5z\" (UID: \"549e448e-4522-41f1-95e2-dffcb399a31c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:11.803372 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.803344 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6lx2\" (UniqueName: \"kubernetes.io/projected/549e448e-4522-41f1-95e2-dffcb399a31c-kube-api-access-b6lx2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c9x5z\" (UID: \"549e448e-4522-41f1-95e2-dffcb399a31c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:11.894746 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.894660 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5hlq\" (UniqueName: \"kubernetes.io/projected/79cbea14-e7aa-448a-9e87-cb4f9005dc31-kube-api-access-w5hlq\") pod \"limitador-operator-controller-manager-859d7b78bd-6mhsf\" (UID: \"79cbea14-e7aa-448a-9e87-cb4f9005dc31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" Jul 10 19:07:11.903972 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.903946 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5hlq\" (UniqueName: \"kubernetes.io/projected/79cbea14-e7aa-448a-9e87-cb4f9005dc31-kube-api-access-w5hlq\") pod \"limitador-operator-controller-manager-859d7b78bd-6mhsf\" (UID: \"79cbea14-e7aa-448a-9e87-cb4f9005dc31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" Jul 10 19:07:11.953209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.953171 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:07:11.995167 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.995133 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt79w\" (UniqueName: \"kubernetes.io/projected/2b83bc62-2c4e-48dd-ba71-61e193ff8e99-kube-api-access-tt79w\") pod \"2b83bc62-2c4e-48dd-ba71-61e193ff8e99\" (UID: \"2b83bc62-2c4e-48dd-ba71-61e193ff8e99\") " Jul 10 19:07:11.997433 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:11.997394 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b83bc62-2c4e-48dd-ba71-61e193ff8e99-kube-api-access-tt79w" (OuterVolumeSpecName: "kube-api-access-tt79w") pod "2b83bc62-2c4e-48dd-ba71-61e193ff8e99" (UID: "2b83bc62-2c4e-48dd-ba71-61e193ff8e99"). InnerVolumeSpecName "kube-api-access-tt79w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:07:12.033401 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.033338 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:12.039522 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.039493 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" Jul 10 19:07:12.096231 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.096198 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt79w\" (UniqueName: \"kubernetes.io/projected/2b83bc62-2c4e-48dd-ba71-61e193ff8e99-kube-api-access-tt79w\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:12.375612 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.375530 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z"] Jul 10 19:07:12.389941 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:07:12.389904 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549e448e_4522_41f1_95e2_dffcb399a31c.slice/crio-d75f12ac43d5ed5980a1485cd5bd36efdbfb8e5875b1eb7d5b28e6c71ac38c5e WatchSource:0}: Error finding container d75f12ac43d5ed5980a1485cd5bd36efdbfb8e5875b1eb7d5b28e6c71ac38c5e: Status 404 returned error can't find the container with id d75f12ac43d5ed5980a1485cd5bd36efdbfb8e5875b1eb7d5b28e6c71ac38c5e Jul 10 19:07:12.436749 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.436719 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf"] Jul 10 19:07:12.441209 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:07:12.441182 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cbea14_e7aa_448a_9e87_cb4f9005dc31.slice/crio-4e6b2c303b32c5418aa0e352c5fb4c34ef59bf29eae35ed206f45543632caa30 WatchSource:0}: Error finding container 4e6b2c303b32c5418aa0e352c5fb4c34ef59bf29eae35ed206f45543632caa30: Status 404 returned error can't find the container with id 4e6b2c303b32c5418aa0e352c5fb4c34ef59bf29eae35ed206f45543632caa30 Jul 10 19:07:12.708565 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.708457 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" event={"ID":"549e448e-4522-41f1-95e2-dffcb399a31c","Type":"ContainerStarted","Data":"d75f12ac43d5ed5980a1485cd5bd36efdbfb8e5875b1eb7d5b28e6c71ac38c5e"} Jul 10 19:07:12.709981 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.709945 2549 generic.go:356] "Generic (PLEG): container finished" podID="2b83bc62-2c4e-48dd-ba71-61e193ff8e99" containerID="49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1" exitCode=0 Jul 10 19:07:12.710121 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.709998 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x948j" Jul 10 19:07:12.710121 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.710051 2549 scope.go:117] "RemoveContainer" containerID="49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1" Jul 10 19:07:12.711719 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.711694 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" event={"ID":"79cbea14-e7aa-448a-9e87-cb4f9005dc31","Type":"ContainerStarted","Data":"3f01fa6da0dc95be0d8d2d410818c41a9f49f6665ac002a991651fbda5b99f9f"} Jul 10 19:07:12.711870 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.711731 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" event={"ID":"79cbea14-e7aa-448a-9e87-cb4f9005dc31","Type":"ContainerStarted","Data":"4e6b2c303b32c5418aa0e352c5fb4c34ef59bf29eae35ed206f45543632caa30"} Jul 10 19:07:12.711870 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.711848 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" Jul 10 19:07:12.719661 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.719645 2549 scope.go:117] "RemoveContainer" containerID="49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1" Jul 10 19:07:12.719987 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:12.719969 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1\": container with ID starting with 49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1 not found: ID does not exist" containerID="49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1" Jul 10 19:07:12.720035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.719996 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1"} err="failed to get container status \"49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1\": rpc error: code = NotFound desc = could not find container \"49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1\": container with ID starting with 49c3280c543218d2f77f718ceaa39c9e9d232838bd141be9786dcf8e43914ba1 not found: ID does not exist" Jul 10 19:07:12.732560 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:12.732516 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" podStartSLOduration=1.732503385 podStartE2EDuration="1.732503385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:12.732169851 +0000 UTC m=+409.483074742" watchObservedRunningTime="2026-07-10 19:07:12.732503385 +0000 UTC m=+409.483408260" Jul 10 19:07:13.805467 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:13.805430 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b83bc62-2c4e-48dd-ba71-61e193ff8e99" path="/var/lib/kubelet/pods/2b83bc62-2c4e-48dd-ba71-61e193ff8e99/volumes" Jul 10 19:07:17.739420 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:17.739281 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" containerName="manager" containerID="cri-o://cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee" gracePeriod=2 Jul 10 19:07:17.741140 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:17.741111 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" event={"ID":"549e448e-4522-41f1-95e2-dffcb399a31c","Type":"ContainerStarted","Data":"fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c"} Jul 10 19:07:17.741254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:17.741230 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:17.741847 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:17.741825 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:17.742860 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:17.742839 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:17.742958 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:17.742880 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:17.743941 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:17.743916 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:17.747925 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:17.747904 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:17.766269 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:17.766229 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:17.766405 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:17.766369 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" podStartSLOduration=2.440483167 podStartE2EDuration="6.766355373s" totalImagesPullingTime="4.325872206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:17.763993299 +0000 UTC m=+414.514898202" watchObservedRunningTime="2026-07-10 19:07:17.766355373 +0000 UTC m=+414.517260324" Jul 10 19:07:17.983979 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:17.983954 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:17.986509 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:17.986485 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:18.054192 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.054105 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b95e08-9442-4613-9477-5c6cbda01ba5-extensions-socket-volume\") pod \"67b95e08-9442-4613-9477-5c6cbda01ba5\" (UID: \"67b95e08-9442-4613-9477-5c6cbda01ba5\") " Jul 10 19:07:18.054192 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.054147 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2dk2\" (UniqueName: \"kubernetes.io/projected/67b95e08-9442-4613-9477-5c6cbda01ba5-kube-api-access-m2dk2\") pod \"67b95e08-9442-4613-9477-5c6cbda01ba5\" (UID: \"67b95e08-9442-4613-9477-5c6cbda01ba5\") " Jul 10 19:07:18.054412 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.054362 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b95e08-9442-4613-9477-5c6cbda01ba5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "67b95e08-9442-4613-9477-5c6cbda01ba5" (UID: "67b95e08-9442-4613-9477-5c6cbda01ba5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:07:18.056394 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.056369 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b95e08-9442-4613-9477-5c6cbda01ba5-kube-api-access-m2dk2" (OuterVolumeSpecName: "kube-api-access-m2dk2") pod "67b95e08-9442-4613-9477-5c6cbda01ba5" (UID: "67b95e08-9442-4613-9477-5c6cbda01ba5"). InnerVolumeSpecName "kube-api-access-m2dk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:07:18.154953 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.154917 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2dk2\" (UniqueName: \"kubernetes.io/projected/67b95e08-9442-4613-9477-5c6cbda01ba5-kube-api-access-m2dk2\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:18.154953 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.154944 2549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b95e08-9442-4613-9477-5c6cbda01ba5-extensions-socket-volume\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:18.747235 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.747199 2549 generic.go:356] "Generic (PLEG): container finished" podID="67b95e08-9442-4613-9477-5c6cbda01ba5" containerID="cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee" exitCode=2 Jul 10 19:07:18.747760 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.747246 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:18.747760 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.747301 2549 scope.go:117] "RemoveContainer" containerID="cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee" Jul 10 19:07:18.749876 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:18.749848 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:18.751594 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.751573 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c655fb948-glxvt" Jul 10 19:07:18.753706 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:18.753681 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:18.756578 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.756561 2549 scope.go:117] "RemoveContainer" containerID="cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee" Jul 10 19:07:18.756848 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:18.756831 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee\": container with ID starting with cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee not found: ID does not exist" containerID="cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee" Jul 10 19:07:18.756900 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.756856 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee"} err="failed to get container status \"cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee\": rpc error: code = NotFound desc = could not find container \"cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee\": container with ID starting with cbacdda9a0e360dc128559ee971f9f0a16d65f9862c43eb1292c71b3ab5306ee not found: ID does not exist" Jul 10 19:07:18.774017 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:18.773983 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:18.811185 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:18.811155 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c776fb86f-z7zp7"] Jul 10 19:07:18.835997 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:18.835921 2549 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-tnfhx\" is forbidden: User \"system:node:ip-10-0-140-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-210.ec2.internal' and this object" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-tnfhx" Jul 10 19:07:19.804650 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:19.804616 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" path="/var/lib/kubelet/pods/67b95e08-9442-4613-9477-5c6cbda01ba5/volumes" Jul 10 19:07:23.721388 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:23.721362 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6mhsf" Jul 10 19:07:24.163941 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.163909 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z"] Jul 10 19:07:24.164147 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.164105 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" podUID="549e448e-4522-41f1-95e2-dffcb399a31c" containerName="manager" containerID="cri-o://fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c" gracePeriod=10 Jul 10 19:07:24.166493 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.166467 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:24.439337 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.439309 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:24.512542 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.512497 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6lx2\" (UniqueName: \"kubernetes.io/projected/549e448e-4522-41f1-95e2-dffcb399a31c-kube-api-access-b6lx2\") pod \"549e448e-4522-41f1-95e2-dffcb399a31c\" (UID: \"549e448e-4522-41f1-95e2-dffcb399a31c\") " Jul 10 19:07:24.512749 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.512611 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/549e448e-4522-41f1-95e2-dffcb399a31c-extensions-socket-volume\") pod \"549e448e-4522-41f1-95e2-dffcb399a31c\" (UID: \"549e448e-4522-41f1-95e2-dffcb399a31c\") " Jul 10 19:07:24.513049 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.513021 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549e448e-4522-41f1-95e2-dffcb399a31c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "549e448e-4522-41f1-95e2-dffcb399a31c" (UID: "549e448e-4522-41f1-95e2-dffcb399a31c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:07:24.514878 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.514856 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549e448e-4522-41f1-95e2-dffcb399a31c-kube-api-access-b6lx2" (OuterVolumeSpecName: "kube-api-access-b6lx2") pod "549e448e-4522-41f1-95e2-dffcb399a31c" (UID: "549e448e-4522-41f1-95e2-dffcb399a31c"). InnerVolumeSpecName "kube-api-access-b6lx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:07:24.613833 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.613768 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6lx2\" (UniqueName: \"kubernetes.io/projected/549e448e-4522-41f1-95e2-dffcb399a31c-kube-api-access-b6lx2\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:24.613833 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.613828 2549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/549e448e-4522-41f1-95e2-dffcb399a31c-extensions-socket-volume\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:24.628905 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.628873 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc"] Jul 10 19:07:24.629216 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.629204 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" containerName="manager" Jul 10 19:07:24.629259 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.629218 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" containerName="manager" Jul 10 19:07:24.629259 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.629231 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549e448e-4522-41f1-95e2-dffcb399a31c" containerName="manager" Jul 10 19:07:24.629259 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.629237 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="549e448e-4522-41f1-95e2-dffcb399a31c" containerName="manager" Jul 10 19:07:24.629351 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.629301 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="549e448e-4522-41f1-95e2-dffcb399a31c" containerName="manager" Jul 10 19:07:24.629351 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.629312 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b95e08-9442-4613-9477-5c6cbda01ba5" containerName="manager" Jul 10 19:07:24.631097 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.631080 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:24.660939 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.660906 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc"] Jul 10 19:07:24.714931 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.714832 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c8xsc\" (UID: \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:24.714931 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.714891 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzxh\" (UniqueName: \"kubernetes.io/projected/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-kube-api-access-xzzxh\") pod \"kuadrant-operator-controller-manager-86878c769f-c8xsc\" (UID: \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:24.779695 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.779655 2549 generic.go:356] "Generic (PLEG): container finished" podID="549e448e-4522-41f1-95e2-dffcb399a31c" containerID="fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c" exitCode=0 Jul 10 19:07:24.780127 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.779703 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" Jul 10 19:07:24.780127 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.779737 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" event={"ID":"549e448e-4522-41f1-95e2-dffcb399a31c","Type":"ContainerDied","Data":"fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c"} Jul 10 19:07:24.780127 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.779808 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z" event={"ID":"549e448e-4522-41f1-95e2-dffcb399a31c","Type":"ContainerDied","Data":"d75f12ac43d5ed5980a1485cd5bd36efdbfb8e5875b1eb7d5b28e6c71ac38c5e"} Jul 10 19:07:24.780127 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.779832 2549 scope.go:117] "RemoveContainer" containerID="fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c" Jul 10 19:07:24.789294 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.789277 2549 scope.go:117] "RemoveContainer" containerID="fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c" Jul 10 19:07:24.789564 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:24.789546 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c\": container with ID starting with fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c not found: ID does not exist" containerID="fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c" Jul 10 19:07:24.789601 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.789576 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c"} err="failed to get container status \"fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c\": rpc error: code = NotFound desc = could not find container \"fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c\": container with ID starting with fd70e463d31d6b57a873b12ac7ce501fcc16343515cd081cec2c83d8ac5ae62c not found: ID does not exist" Jul 10 19:07:24.802552 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.802522 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z"] Jul 10 19:07:24.815769 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.815738 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c8xsc\" (UID: \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:24.815948 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.815822 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzxh\" (UniqueName: \"kubernetes.io/projected/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-kube-api-access-xzzxh\") pod \"kuadrant-operator-controller-manager-86878c769f-c8xsc\" (UID: \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:24.816189 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.816162 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c8xsc\" (UID: \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:24.817478 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.817458 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c9x5z"] Jul 10 19:07:24.830276 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.830249 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzxh\" (UniqueName: \"kubernetes.io/projected/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-kube-api-access-xzzxh\") pod \"kuadrant-operator-controller-manager-86878c769f-c8xsc\" (UID: \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:24.941346 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:24.941300 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:25.092118 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:25.092095 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc"] Jul 10 19:07:25.095270 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:07:25.095240 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fe6cad_5bdf_4df8_acdd_404acc8e55aa.slice/crio-3ec4a4312fc7935006a160793b98451782fd867ba6e5b514bb916ca4f4ed193f WatchSource:0}: Error finding container 3ec4a4312fc7935006a160793b98451782fd867ba6e5b514bb916ca4f4ed193f: Status 404 returned error can't find the container with id 3ec4a4312fc7935006a160793b98451782fd867ba6e5b514bb916ca4f4ed193f Jul 10 19:07:25.786496 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:25.786456 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" event={"ID":"65fe6cad-5bdf-4df8-acdd-404acc8e55aa","Type":"ContainerStarted","Data":"c41c85499880b97a4a5b98ece383e5072bcaed3cd597bda395691c621e00c9e5"} Jul 10 19:07:25.786984 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:25.786633 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:25.786984 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:25.786672 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" event={"ID":"65fe6cad-5bdf-4df8-acdd-404acc8e55aa","Type":"ContainerStarted","Data":"3ec4a4312fc7935006a160793b98451782fd867ba6e5b514bb916ca4f4ed193f"} Jul 10 19:07:25.804249 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:25.804215 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549e448e-4522-41f1-95e2-dffcb399a31c" path="/var/lib/kubelet/pods/549e448e-4522-41f1-95e2-dffcb399a31c/volumes" Jul 10 19:07:25.809973 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:25.809923 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" podStartSLOduration=1.809903382 podStartE2EDuration="1.809903382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:25.807666175 +0000 UTC m=+422.558571073" watchObservedRunningTime="2026-07-10 19:07:25.809903382 +0000 UTC m=+422.560808258" Jul 10 19:07:36.796120 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:36.796087 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:07:43.834185 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:43.834132 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c776fb86f-z7zp7" podUID="18347256-714d-40ff-ab18-f2b259151f70" containerName="console" containerID="cri-o://ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01" gracePeriod=15 Jul 10 19:07:44.087308 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.087247 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:07:44.182283 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182245 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-console-config\") pod \"18347256-714d-40ff-ab18-f2b259151f70\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " Jul 10 19:07:44.182469 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182295 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-trusted-ca-bundle\") pod \"18347256-714d-40ff-ab18-f2b259151f70\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " Jul 10 19:07:44.182469 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182320 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-oauth-config\") pod \"18347256-714d-40ff-ab18-f2b259151f70\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " Jul 10 19:07:44.182469 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182354 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-serving-cert\") pod \"18347256-714d-40ff-ab18-f2b259151f70\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " Jul 10 19:07:44.182469 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182379 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-oauth-serving-cert\") pod \"18347256-714d-40ff-ab18-f2b259151f70\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " Jul 10 19:07:44.182469 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182408 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4rx6\" (UniqueName: \"kubernetes.io/projected/18347256-714d-40ff-ab18-f2b259151f70-kube-api-access-c4rx6\") pod \"18347256-714d-40ff-ab18-f2b259151f70\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " Jul 10 19:07:44.182469 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182438 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-service-ca\") pod \"18347256-714d-40ff-ab18-f2b259151f70\" (UID: \"18347256-714d-40ff-ab18-f2b259151f70\") " Jul 10 19:07:44.182849 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182814 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-console-config" (OuterVolumeSpecName: "console-config") pod "18347256-714d-40ff-ab18-f2b259151f70" (UID: "18347256-714d-40ff-ab18-f2b259151f70"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:07:44.182969 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182949 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "18347256-714d-40ff-ab18-f2b259151f70" (UID: "18347256-714d-40ff-ab18-f2b259151f70"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:07:44.183032 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.182988 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "18347256-714d-40ff-ab18-f2b259151f70" (UID: "18347256-714d-40ff-ab18-f2b259151f70"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:07:44.183032 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.183003 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-service-ca" (OuterVolumeSpecName: "service-ca") pod "18347256-714d-40ff-ab18-f2b259151f70" (UID: "18347256-714d-40ff-ab18-f2b259151f70"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:07:44.184721 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.184692 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "18347256-714d-40ff-ab18-f2b259151f70" (UID: "18347256-714d-40ff-ab18-f2b259151f70"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:07:44.184806 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.184743 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18347256-714d-40ff-ab18-f2b259151f70-kube-api-access-c4rx6" (OuterVolumeSpecName: "kube-api-access-c4rx6") pod "18347256-714d-40ff-ab18-f2b259151f70" (UID: "18347256-714d-40ff-ab18-f2b259151f70"). InnerVolumeSpecName "kube-api-access-c4rx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:07:44.184884 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.184860 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "18347256-714d-40ff-ab18-f2b259151f70" (UID: "18347256-714d-40ff-ab18-f2b259151f70"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:07:44.283132 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.283097 2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-serving-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:44.283132 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.283125 2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-oauth-serving-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:44.283132 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.283134 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4rx6\" (UniqueName: \"kubernetes.io/projected/18347256-714d-40ff-ab18-f2b259151f70-kube-api-access-c4rx6\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:44.283358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.283144 2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:44.283358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.283153 2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-console-config\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:44.283358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.283160 2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18347256-714d-40ff-ab18-f2b259151f70-trusted-ca-bundle\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:44.283358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.283168 2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18347256-714d-40ff-ab18-f2b259151f70-console-oauth-config\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:07:44.877487 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.877401 2549 generic.go:356] "Generic (PLEG): container finished" podID="18347256-714d-40ff-ab18-f2b259151f70" containerID="ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01" exitCode=0 Jul 10 19:07:44.877487 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.877441 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c776fb86f-z7zp7" event={"ID":"18347256-714d-40ff-ab18-f2b259151f70","Type":"ContainerDied","Data":"ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01"} Jul 10 19:07:44.877487 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.877458 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c776fb86f-z7zp7" Jul 10 19:07:44.877487 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.877480 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c776fb86f-z7zp7" event={"ID":"18347256-714d-40ff-ab18-f2b259151f70","Type":"ContainerDied","Data":"0dc3c5bd7ddabbd7f124e559e851937a94fe82a34b5fcee2197e6646181b7bad"} Jul 10 19:07:44.878017 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.877501 2549 scope.go:117] "RemoveContainer" containerID="ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01" Jul 10 19:07:44.888480 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.888458 2549 scope.go:117] "RemoveContainer" containerID="ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01" Jul 10 19:07:44.888752 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:07:44.888731 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01\": container with ID starting with ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01 not found: ID does not exist" containerID="ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01" Jul 10 19:07:44.888841 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.888765 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01"} err="failed to get container status \"ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01\": rpc error: code = NotFound desc = could not find container \"ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01\": container with ID starting with ed5c9a34a433609b363c95126db1450b4f317361fee4cbcb6928492d8e0c5a01 not found: ID does not exist" Jul 10 19:07:44.901601 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.901579 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c776fb86f-z7zp7"] Jul 10 19:07:44.905231 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:44.905202 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c776fb86f-z7zp7"] Jul 10 19:07:45.804216 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:45.804179 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18347256-714d-40ff-ab18-f2b259151f70" path="/var/lib/kubelet/pods/18347256-714d-40ff-ab18-f2b259151f70/volumes" Jul 10 19:07:56.791035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.791000 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-7q56h"] Jul 10 19:07:56.791511 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.791337 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18347256-714d-40ff-ab18-f2b259151f70" containerName="console" Jul 10 19:07:56.791511 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.791349 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="18347256-714d-40ff-ab18-f2b259151f70" containerName="console" Jul 10 19:07:56.791511 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.791412 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="18347256-714d-40ff-ab18-f2b259151f70" containerName="console" Jul 10 19:07:56.793428 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.793412 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7q56h" Jul 10 19:07:56.796024 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.796002 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jdvs4\"" Jul 10 19:07:56.811256 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.811229 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7q56h"] Jul 10 19:07:56.890128 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.890086 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89kdv\" (UniqueName: \"kubernetes.io/projected/2eb3fc2a-ff66-4d46-b061-745078865db4-kube-api-access-89kdv\") pod \"authorino-75d556bd77-7q56h\" (UID: \"2eb3fc2a-ff66-4d46-b061-745078865db4\") " pod="kuadrant-system/authorino-75d556bd77-7q56h" Jul 10 19:07:56.991556 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:56.991521 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89kdv\" (UniqueName: \"kubernetes.io/projected/2eb3fc2a-ff66-4d46-b061-745078865db4-kube-api-access-89kdv\") pod \"authorino-75d556bd77-7q56h\" (UID: \"2eb3fc2a-ff66-4d46-b061-745078865db4\") " pod="kuadrant-system/authorino-75d556bd77-7q56h" Jul 10 19:07:57.002998 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:57.002970 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kdv\" (UniqueName: \"kubernetes.io/projected/2eb3fc2a-ff66-4d46-b061-745078865db4-kube-api-access-89kdv\") pod \"authorino-75d556bd77-7q56h\" (UID: \"2eb3fc2a-ff66-4d46-b061-745078865db4\") " pod="kuadrant-system/authorino-75d556bd77-7q56h" Jul 10 19:07:57.102690 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:57.102656 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7q56h" Jul 10 19:07:57.236144 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:57.236118 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7q56h"] Jul 10 19:07:57.238647 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:07:57.238604 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eb3fc2a_ff66_4d46_b061_745078865db4.slice/crio-9798023b50b9d4f74a88951ce6c1778c76d480e12dfecbcdcc1639d4575d004f WatchSource:0}: Error finding container 9798023b50b9d4f74a88951ce6c1778c76d480e12dfecbcdcc1639d4575d004f: Status 404 returned error can't find the container with id 9798023b50b9d4f74a88951ce6c1778c76d480e12dfecbcdcc1639d4575d004f Jul 10 19:07:57.944753 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:07:57.944715 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7q56h" event={"ID":"2eb3fc2a-ff66-4d46-b061-745078865db4","Type":"ContainerStarted","Data":"9798023b50b9d4f74a88951ce6c1778c76d480e12dfecbcdcc1639d4575d004f"} Jul 10 19:08:00.963838 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:00.963774 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7q56h" event={"ID":"2eb3fc2a-ff66-4d46-b061-745078865db4","Type":"ContainerStarted","Data":"d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495"} Jul 10 19:08:00.979985 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:00.979915 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-7q56h" podStartSLOduration=2.163188265 podStartE2EDuration="4.979895401s" totalImagesPullingTime="2.816707136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:00.979411715 +0000 UTC m=+457.730316571" watchObservedRunningTime="2026-07-10 19:08:00.979895401 +0000 UTC m=+457.730800277" Jul 10 19:08:29.521564 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:29.521527 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7q56h"] Jul 10 19:08:29.522131 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:29.521713 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-7q56h" podUID="2eb3fc2a-ff66-4d46-b061-745078865db4" containerName="authorino" containerID="cri-o://d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495" gracePeriod=30 Jul 10 19:08:29.772696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:29.772636 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7q56h" Jul 10 19:08:29.885055 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:29.885022 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89kdv\" (UniqueName: \"kubernetes.io/projected/2eb3fc2a-ff66-4d46-b061-745078865db4-kube-api-access-89kdv\") pod \"2eb3fc2a-ff66-4d46-b061-745078865db4\" (UID: \"2eb3fc2a-ff66-4d46-b061-745078865db4\") " Jul 10 19:08:29.887197 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:29.887172 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb3fc2a-ff66-4d46-b061-745078865db4-kube-api-access-89kdv" (OuterVolumeSpecName: "kube-api-access-89kdv") pod "2eb3fc2a-ff66-4d46-b061-745078865db4" (UID: "2eb3fc2a-ff66-4d46-b061-745078865db4"). InnerVolumeSpecName "kube-api-access-89kdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:08:29.986383 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:29.986348 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89kdv\" (UniqueName: \"kubernetes.io/projected/2eb3fc2a-ff66-4d46-b061-745078865db4-kube-api-access-89kdv\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:08:30.097567 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.097533 2549 generic.go:356] "Generic (PLEG): container finished" podID="2eb3fc2a-ff66-4d46-b061-745078865db4" containerID="d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495" exitCode=0 Jul 10 19:08:30.097747 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.097583 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7q56h" Jul 10 19:08:30.097747 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.097587 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7q56h" event={"ID":"2eb3fc2a-ff66-4d46-b061-745078865db4","Type":"ContainerDied","Data":"d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495"} Jul 10 19:08:30.097747 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.097693 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7q56h" event={"ID":"2eb3fc2a-ff66-4d46-b061-745078865db4","Type":"ContainerDied","Data":"9798023b50b9d4f74a88951ce6c1778c76d480e12dfecbcdcc1639d4575d004f"} Jul 10 19:08:30.097747 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.097714 2549 scope.go:117] "RemoveContainer" containerID="d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495" Jul 10 19:08:30.106628 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.106611 2549 scope.go:117] "RemoveContainer" containerID="d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495" Jul 10 19:08:30.106966 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:08:30.106915 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495\": container with ID starting with d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495 not found: ID does not exist" containerID="d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495" Jul 10 19:08:30.106966 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.106951 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495"} err="failed to get container status \"d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495\": rpc error: code = NotFound desc = could not find container \"d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495\": container with ID starting with d72a9b3455947e095611f4287e3f98666edce75c0f28859b540531e6788aa495 not found: ID does not exist" Jul 10 19:08:30.121254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.121230 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7q56h"] Jul 10 19:08:30.130981 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:30.130956 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7q56h"] Jul 10 19:08:31.804548 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:31.804512 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb3fc2a-ff66-4d46-b061-745078865db4" path="/var/lib/kubelet/pods/2eb3fc2a-ff66-4d46-b061-745078865db4/volumes" Jul 10 19:08:34.098613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.098581 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58fbff7847-78zdb"] Jul 10 19:08:34.099018 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.098926 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb3fc2a-ff66-4d46-b061-745078865db4" containerName="authorino" Jul 10 19:08:34.099018 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.098937 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb3fc2a-ff66-4d46-b061-745078865db4" containerName="authorino" Jul 10 19:08:34.099018 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.099005 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb3fc2a-ff66-4d46-b061-745078865db4" containerName="authorino" Jul 10 19:08:34.103037 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.103017 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:34.106117 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.106082 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 19:08:34.106254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.106087 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 19:08:34.106254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.106163 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l6jrh\"" Jul 10 19:08:34.111989 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.111962 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58fbff7847-78zdb"] Jul 10 19:08:34.118826 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.118773 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert\") pod \"maas-controller-58fbff7847-78zdb\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:34.118826 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.118823 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9mf\" (UniqueName: \"kubernetes.io/projected/b1d669e8-b3c4-43fc-8715-9f42bc68e866-kube-api-access-5h9mf\") pod \"maas-controller-58fbff7847-78zdb\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:34.219641 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.219600 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert\") pod \"maas-controller-58fbff7847-78zdb\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:34.219641 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.219642 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9mf\" (UniqueName: \"kubernetes.io/projected/b1d669e8-b3c4-43fc-8715-9f42bc68e866-kube-api-access-5h9mf\") pod \"maas-controller-58fbff7847-78zdb\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:34.219879 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:08:34.219765 2549 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 19:08:34.219879 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:08:34.219858 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert podName:b1d669e8-b3c4-43fc-8715-9f42bc68e866 nodeName:}" failed. No retries permitted until 2026-07-10 19:08:34.719841541 +0000 UTC m=+491.470746392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert") pod "maas-controller-58fbff7847-78zdb" (UID: "b1d669e8-b3c4-43fc-8715-9f42bc68e866") : secret "maas-controller-webhook-cert" not found Jul 10 19:08:34.231655 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.231621 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9mf\" (UniqueName: \"kubernetes.io/projected/b1d669e8-b3c4-43fc-8715-9f42bc68e866-kube-api-access-5h9mf\") pod \"maas-controller-58fbff7847-78zdb\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:34.288209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.288175 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-98f7fc48c-b77sg"] Jul 10 19:08:34.290490 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.290473 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:34.301333 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.301306 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-98f7fc48c-b77sg"] Jul 10 19:08:34.320354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.320319 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6l2h\" (UniqueName: \"kubernetes.io/projected/da6e4e8b-40f7-49f4-a82f-e75a35399b64-kube-api-access-g6l2h\") pod \"maas-controller-98f7fc48c-b77sg\" (UID: \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\") " pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:34.320480 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.320401 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6e4e8b-40f7-49f4-a82f-e75a35399b64-webhook-cert\") pod \"maas-controller-98f7fc48c-b77sg\" (UID: \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\") " pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:34.421071 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.420964 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6e4e8b-40f7-49f4-a82f-e75a35399b64-webhook-cert\") pod \"maas-controller-98f7fc48c-b77sg\" (UID: \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\") " pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:34.421071 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.421027 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6l2h\" (UniqueName: \"kubernetes.io/projected/da6e4e8b-40f7-49f4-a82f-e75a35399b64-kube-api-access-g6l2h\") pod \"maas-controller-98f7fc48c-b77sg\" (UID: \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\") " pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:34.423826 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.423771 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6e4e8b-40f7-49f4-a82f-e75a35399b64-webhook-cert\") pod \"maas-controller-98f7fc48c-b77sg\" (UID: \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\") " pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:34.432087 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.432053 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6l2h\" (UniqueName: \"kubernetes.io/projected/da6e4e8b-40f7-49f4-a82f-e75a35399b64-kube-api-access-g6l2h\") pod \"maas-controller-98f7fc48c-b77sg\" (UID: \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\") " pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:34.601325 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.601289 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:34.723696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.723663 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert\") pod \"maas-controller-58fbff7847-78zdb\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:34.726257 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.726235 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert\") pod \"maas-controller-58fbff7847-78zdb\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:34.741356 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:34.741330 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-98f7fc48c-b77sg"] Jul 10 19:08:34.744744 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:08:34.744717 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6e4e8b_40f7_49f4_a82f_e75a35399b64.slice/crio-e0761a3484987bdd8b5f9b267377132a4cfc0c526efb32cb54c5601bd007a315 WatchSource:0}: Error finding container e0761a3484987bdd8b5f9b267377132a4cfc0c526efb32cb54c5601bd007a315: Status 404 returned error can't find the container with id e0761a3484987bdd8b5f9b267377132a4cfc0c526efb32cb54c5601bd007a315 Jul 10 19:08:35.014699 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:35.014610 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:35.125560 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:35.125529 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98f7fc48c-b77sg" event={"ID":"da6e4e8b-40f7-49f4-a82f-e75a35399b64","Type":"ContainerStarted","Data":"e0761a3484987bdd8b5f9b267377132a4cfc0c526efb32cb54c5601bd007a315"} Jul 10 19:08:35.152767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:35.152742 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58fbff7847-78zdb"] Jul 10 19:08:35.154491 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:08:35.154463 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d669e8_b3c4_43fc_8715_9f42bc68e866.slice/crio-a81aa11cb12d131f4be34d5de6e535b3c2ab570d39ff65e7181675befd1f5851 WatchSource:0}: Error finding container a81aa11cb12d131f4be34d5de6e535b3c2ab570d39ff65e7181675befd1f5851: Status 404 returned error can't find the container with id a81aa11cb12d131f4be34d5de6e535b3c2ab570d39ff65e7181675befd1f5851 Jul 10 19:08:36.139595 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:36.139505 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fbff7847-78zdb" event={"ID":"b1d669e8-b3c4-43fc-8715-9f42bc68e866","Type":"ContainerStarted","Data":"a81aa11cb12d131f4be34d5de6e535b3c2ab570d39ff65e7181675befd1f5851"} Jul 10 19:08:37.148380 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:37.148337 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fbff7847-78zdb" event={"ID":"b1d669e8-b3c4-43fc-8715-9f42bc68e866","Type":"ContainerStarted","Data":"e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c"} Jul 10 19:08:37.149059 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:37.148426 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:37.150859 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:37.150831 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98f7fc48c-b77sg" event={"ID":"da6e4e8b-40f7-49f4-a82f-e75a35399b64","Type":"ContainerStarted","Data":"25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b"} Jul 10 19:08:37.151353 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:37.151332 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:37.194514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:37.194421 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58fbff7847-78zdb" podStartSLOduration=1.396997123 podStartE2EDuration="3.194404376s" totalImagesPullingTime="1.797407253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:37.190461461 +0000 UTC m=+493.941366385" watchObservedRunningTime="2026-07-10 19:08:37.194404376 +0000 UTC m=+493.945309252" Jul 10 19:08:37.228447 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:37.228396 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-98f7fc48c-b77sg" podStartSLOduration=1.03075715 podStartE2EDuration="3.228378603s" totalImagesPullingTime="2.197621453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:37.226683344 +0000 UTC m=+493.977588219" watchObservedRunningTime="2026-07-10 19:08:37.228378603 +0000 UTC m=+493.979283477" Jul 10 19:08:48.163344 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.163315 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:08:48.163775 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.163384 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:48.230605 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.230571 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58fbff7847-78zdb"] Jul 10 19:08:48.230851 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.230823 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58fbff7847-78zdb" podUID="b1d669e8-b3c4-43fc-8715-9f42bc68e866" containerName="manager" containerID="cri-o://e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c" gracePeriod=30 Jul 10 19:08:48.486630 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.486608 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:48.546867 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.546831 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h9mf\" (UniqueName: \"kubernetes.io/projected/b1d669e8-b3c4-43fc-8715-9f42bc68e866-kube-api-access-5h9mf\") pod \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " Jul 10 19:08:48.547027 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.546920 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert\") pod \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\" (UID: \"b1d669e8-b3c4-43fc-8715-9f42bc68e866\") " Jul 10 19:08:48.549150 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.549126 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b1d669e8-b3c4-43fc-8715-9f42bc68e866" (UID: "b1d669e8-b3c4-43fc-8715-9f42bc68e866"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:08:48.549259 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.549170 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d669e8-b3c4-43fc-8715-9f42bc68e866-kube-api-access-5h9mf" (OuterVolumeSpecName: "kube-api-access-5h9mf") pod "b1d669e8-b3c4-43fc-8715-9f42bc68e866" (UID: "b1d669e8-b3c4-43fc-8715-9f42bc68e866"). InnerVolumeSpecName "kube-api-access-5h9mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:08:48.648254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.648224 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h9mf\" (UniqueName: \"kubernetes.io/projected/b1d669e8-b3c4-43fc-8715-9f42bc68e866-kube-api-access-5h9mf\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:08:48.648254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:48.648249 2549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1d669e8-b3c4-43fc-8715-9f42bc68e866-webhook-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:08:49.211942 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.211909 2549 generic.go:356] "Generic (PLEG): container finished" podID="b1d669e8-b3c4-43fc-8715-9f42bc68e866" containerID="e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c" exitCode=0 Jul 10 19:08:49.212353 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.211959 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fbff7847-78zdb" Jul 10 19:08:49.212353 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.211993 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fbff7847-78zdb" event={"ID":"b1d669e8-b3c4-43fc-8715-9f42bc68e866","Type":"ContainerDied","Data":"e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c"} Jul 10 19:08:49.212353 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.212044 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fbff7847-78zdb" event={"ID":"b1d669e8-b3c4-43fc-8715-9f42bc68e866","Type":"ContainerDied","Data":"a81aa11cb12d131f4be34d5de6e535b3c2ab570d39ff65e7181675befd1f5851"} Jul 10 19:08:49.212353 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.212069 2549 scope.go:117] "RemoveContainer" containerID="e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c" Jul 10 19:08:49.221341 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.221324 2549 scope.go:117] "RemoveContainer" containerID="e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c" Jul 10 19:08:49.221585 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:08:49.221566 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c\": container with ID starting with e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c not found: ID does not exist" containerID="e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c" Jul 10 19:08:49.221648 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.221596 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c"} err="failed to get container status \"e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c\": rpc error: code = NotFound desc = could not find container \"e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c\": container with ID starting with e52bd90431813d6e1264435e16cacce0b92802dd7c1319c13659d9c4e9235a5c not found: ID does not exist" Jul 10 19:08:49.233467 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.233448 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58fbff7847-78zdb"] Jul 10 19:08:49.243546 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.243521 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58fbff7847-78zdb"] Jul 10 19:08:49.804642 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:08:49.804607 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d669e8-b3c4-43fc-8715-9f42bc68e866" path="/var/lib/kubelet/pods/b1d669e8-b3c4-43fc-8715-9f42bc68e866/volumes" Jul 10 19:09:09.203565 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.203527 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6695c44845-thjc5"] Jul 10 19:09:09.203980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.203923 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d669e8-b3c4-43fc-8715-9f42bc68e866" containerName="manager" Jul 10 19:09:09.203980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.203934 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d669e8-b3c4-43fc-8715-9f42bc68e866" containerName="manager" Jul 10 19:09:09.204048 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.203999 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d669e8-b3c4-43fc-8715-9f42bc68e866" containerName="manager" Jul 10 19:09:09.208368 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.208352 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.211229 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.211178 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 19:09:09.211229 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.211219 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 19:09:09.211442 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.211224 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pj288\"" Jul 10 19:09:09.211442 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.211290 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 19:09:09.222364 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.222338 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6695c44845-thjc5"] Jul 10 19:09:09.328679 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.328645 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07411a59-ce66-4534-8a33-80da0ea8e051-openshift-service-ca\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.328875 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.328688 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmll\" (UniqueName: \"kubernetes.io/projected/07411a59-ce66-4534-8a33-80da0ea8e051-kube-api-access-qdmll\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.328875 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.328713 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07411a59-ce66-4534-8a33-80da0ea8e051-maas-api-tls\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.429569 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.429533 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07411a59-ce66-4534-8a33-80da0ea8e051-openshift-service-ca\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.429718 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.429574 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmll\" (UniqueName: \"kubernetes.io/projected/07411a59-ce66-4534-8a33-80da0ea8e051-kube-api-access-qdmll\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.429718 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.429690 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07411a59-ce66-4534-8a33-80da0ea8e051-maas-api-tls\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.430261 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.430233 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07411a59-ce66-4534-8a33-80da0ea8e051-openshift-service-ca\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.432177 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.432154 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07411a59-ce66-4534-8a33-80da0ea8e051-maas-api-tls\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.441998 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.441974 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmll\" (UniqueName: \"kubernetes.io/projected/07411a59-ce66-4534-8a33-80da0ea8e051-kube-api-access-qdmll\") pod \"maas-api-6695c44845-thjc5\" (UID: \"07411a59-ce66-4534-8a33-80da0ea8e051\") " pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.518779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.518701 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:09.678902 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.678868 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6695c44845-thjc5"] Jul 10 19:09:09.683044 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:09:09.683016 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07411a59_ce66_4534_8a33_80da0ea8e051.slice/crio-43eee48d66bbe9b31dbe675784420903c7103b354763123b8046f4f2660bd740 WatchSource:0}: Error finding container 43eee48d66bbe9b31dbe675784420903c7103b354763123b8046f4f2660bd740: Status 404 returned error can't find the container with id 43eee48d66bbe9b31dbe675784420903c7103b354763123b8046f4f2660bd740 Jul 10 19:09:09.758484 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.758450 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ph9cl"] Jul 10 19:09:09.762902 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.762881 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:09.766378 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.766358 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 19:09:09.767890 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.767873 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xsgpq"] Jul 10 19:09:09.772839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.772818 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ph9cl"] Jul 10 19:09:09.772950 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.772887 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:09.775383 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.775362 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-t6r7s\"" Jul 10 19:09:09.780937 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.780914 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xsgpq"] Jul 10 19:09:09.934854 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.934820 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f52dw\" (UniqueName: \"kubernetes.io/projected/5b1ea410-9c31-4085-ad65-c38124158b9e-kube-api-access-f52dw\") pod \"payload-processing-58f86c7798-xsgpq\" (UID: \"5b1ea410-9c31-4085-ad65-c38124158b9e\") " pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:09.935047 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.934893 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183ef325-6e3f-4700-8913-1de86302ea0a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ph9cl\" (UID: \"183ef325-6e3f-4700-8913-1de86302ea0a\") " pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:09.935047 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.934998 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76lv8\" (UniqueName: \"kubernetes.io/projected/183ef325-6e3f-4700-8913-1de86302ea0a-kube-api-access-76lv8\") pod \"payload-pre-processing-7964457696-ph9cl\" (UID: \"183ef325-6e3f-4700-8913-1de86302ea0a\") " pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:09.935047 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:09.935045 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1ea410-9c31-4085-ad65-c38124158b9e-plugins-config-volume\") pod \"payload-processing-58f86c7798-xsgpq\" (UID: \"5b1ea410-9c31-4085-ad65-c38124158b9e\") " pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:10.036164 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.036061 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f52dw\" (UniqueName: \"kubernetes.io/projected/5b1ea410-9c31-4085-ad65-c38124158b9e-kube-api-access-f52dw\") pod \"payload-processing-58f86c7798-xsgpq\" (UID: \"5b1ea410-9c31-4085-ad65-c38124158b9e\") " pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:10.036164 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.036126 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183ef325-6e3f-4700-8913-1de86302ea0a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ph9cl\" (UID: \"183ef325-6e3f-4700-8913-1de86302ea0a\") " pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:10.036454 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.036205 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76lv8\" (UniqueName: \"kubernetes.io/projected/183ef325-6e3f-4700-8913-1de86302ea0a-kube-api-access-76lv8\") pod \"payload-pre-processing-7964457696-ph9cl\" (UID: \"183ef325-6e3f-4700-8913-1de86302ea0a\") " pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:10.036454 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.036259 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1ea410-9c31-4085-ad65-c38124158b9e-plugins-config-volume\") pod \"payload-processing-58f86c7798-xsgpq\" (UID: \"5b1ea410-9c31-4085-ad65-c38124158b9e\") " pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:10.037072 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.037046 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/183ef325-6e3f-4700-8913-1de86302ea0a-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ph9cl\" (UID: \"183ef325-6e3f-4700-8913-1de86302ea0a\") " pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:10.037177 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.037084 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1ea410-9c31-4085-ad65-c38124158b9e-plugins-config-volume\") pod \"payload-processing-58f86c7798-xsgpq\" (UID: \"5b1ea410-9c31-4085-ad65-c38124158b9e\") " pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:10.047654 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.047629 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f52dw\" (UniqueName: \"kubernetes.io/projected/5b1ea410-9c31-4085-ad65-c38124158b9e-kube-api-access-f52dw\") pod \"payload-processing-58f86c7798-xsgpq\" (UID: \"5b1ea410-9c31-4085-ad65-c38124158b9e\") " pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:10.048281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.048258 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76lv8\" (UniqueName: \"kubernetes.io/projected/183ef325-6e3f-4700-8913-1de86302ea0a-kube-api-access-76lv8\") pod \"payload-pre-processing-7964457696-ph9cl\" (UID: \"183ef325-6e3f-4700-8913-1de86302ea0a\") " pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:10.075711 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.075681 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:10.087101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.087078 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:10.233350 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.233319 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ph9cl"] Jul 10 19:09:10.234577 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:09:10.234550 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183ef325_6e3f_4700_8913_1de86302ea0a.slice/crio-da903ac7cb09e29ffec83d63a7a2b658b0ac7935f96108bf10e48588045e5b3c WatchSource:0}: Error finding container da903ac7cb09e29ffec83d63a7a2b658b0ac7935f96108bf10e48588045e5b3c: Status 404 returned error can't find the container with id da903ac7cb09e29ffec83d63a7a2b658b0ac7935f96108bf10e48588045e5b3c Jul 10 19:09:10.256427 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.256403 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xsgpq"] Jul 10 19:09:10.258686 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:09:10.258650 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1ea410_9c31_4085_ad65_c38124158b9e.slice/crio-4d3d64f60cb63f41d88f38e71d7965ff6bc58d808c2d80874ca9cc86f6b3a3cc WatchSource:0}: Error finding container 4d3d64f60cb63f41d88f38e71d7965ff6bc58d808c2d80874ca9cc86f6b3a3cc: Status 404 returned error can't find the container with id 4d3d64f60cb63f41d88f38e71d7965ff6bc58d808c2d80874ca9cc86f6b3a3cc Jul 10 19:09:10.310523 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.310437 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" event={"ID":"183ef325-6e3f-4700-8913-1de86302ea0a","Type":"ContainerStarted","Data":"da903ac7cb09e29ffec83d63a7a2b658b0ac7935f96108bf10e48588045e5b3c"} Jul 10 19:09:10.311717 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.311688 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" event={"ID":"07411a59-ce66-4534-8a33-80da0ea8e051","Type":"ContainerStarted","Data":"43eee48d66bbe9b31dbe675784420903c7103b354763123b8046f4f2660bd740"} Jul 10 19:09:10.312848 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:10.312815 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" event={"ID":"5b1ea410-9c31-4085-ad65-c38124158b9e","Type":"ContainerStarted","Data":"4d3d64f60cb63f41d88f38e71d7965ff6bc58d808c2d80874ca9cc86f6b3a3cc"} Jul 10 19:09:12.324170 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.324126 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" event={"ID":"5b1ea410-9c31-4085-ad65-c38124158b9e","Type":"ContainerStarted","Data":"4bd4f58bc742f2bf0a7e9250dfb69381656a772f84aeaa2aebbee94bd39ff508"} Jul 10 19:09:12.324627 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.324286 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:12.325678 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.325652 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" event={"ID":"183ef325-6e3f-4700-8913-1de86302ea0a","Type":"ContainerStarted","Data":"4da77245edefe54c2ef92fd62c38fbb4749ff97a9e52f9e5a597202437c228b7"} Jul 10 19:09:12.325828 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.325699 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:12.327090 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.327069 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" event={"ID":"07411a59-ce66-4534-8a33-80da0ea8e051","Type":"ContainerStarted","Data":"f298e03d1ec4e265bb78f558afc4c98e932e987ffe79fc6938d3b57a417d52ce"} Jul 10 19:09:12.327195 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.327179 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:12.344355 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.344308 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" podStartSLOduration=1.788315265 podStartE2EDuration="3.344294499s" totalImagesPullingTime="1.555979234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:12.341970807 +0000 UTC m=+529.092875680" watchObservedRunningTime="2026-07-10 19:09:12.344294499 +0000 UTC m=+529.095199373" Jul 10 19:09:12.362733 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.362689 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" podStartSLOduration=1.697414382 podStartE2EDuration="3.362673957s" totalImagesPullingTime="1.665259575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:12.361539696 +0000 UTC m=+529.112444584" watchObservedRunningTime="2026-07-10 19:09:12.362673957 +0000 UTC m=+529.113578831" Jul 10 19:09:12.379120 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:12.379072 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" podStartSLOduration=1.801398502 podStartE2EDuration="3.379054654s" totalImagesPullingTime="1.577656152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:12.37861311 +0000 UTC m=+529.129518009" watchObservedRunningTime="2026-07-10 19:09:12.379054654 +0000 UTC m=+529.129959529" Jul 10 19:09:18.343033 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:18.342950 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6695c44845-thjc5" Jul 10 19:09:23.339057 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:23.339023 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-xsgpq" Jul 10 19:09:23.339547 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:23.339108 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-ph9cl" Jul 10 19:09:52.296540 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.296497 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn"] Jul 10 19:09:52.300261 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.300238 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.303907 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.303882 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vc5dn\"" Jul 10 19:09:52.304035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.303907 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 19:09:52.304035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.303958 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 19:09:52.304035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.303995 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 19:09:52.313927 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.313898 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn"] Jul 10 19:09:52.406966 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.406927 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580fe534-ceef-40f1-9d9a-a46fe84457b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.407150 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.406979 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.407150 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.407076 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.407150 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.407108 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzmw\" (UniqueName: \"kubernetes.io/projected/580fe534-ceef-40f1-9d9a-a46fe84457b9-kube-api-access-5jzmw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.407150 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.407130 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.407304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.407265 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.508474 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.508433 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.508659 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.508495 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580fe534-ceef-40f1-9d9a-a46fe84457b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.508659 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.508521 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.508659 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.508569 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.508659 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.508592 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzmw\" (UniqueName: \"kubernetes.io/projected/580fe534-ceef-40f1-9d9a-a46fe84457b9-kube-api-access-5jzmw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.508659 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.508616 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.508972 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.508926 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.508972 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.508941 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.509071 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.509010 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.511119 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.511089 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/580fe534-ceef-40f1-9d9a-a46fe84457b9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.511308 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.511289 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/580fe534-ceef-40f1-9d9a-a46fe84457b9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.519022 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.519001 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzmw\" (UniqueName: \"kubernetes.io/projected/580fe534-ceef-40f1-9d9a-a46fe84457b9-kube-api-access-5jzmw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-2hcvn\" (UID: \"580fe534-ceef-40f1-9d9a-a46fe84457b9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.610811 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.610774 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:09:52.756092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:52.756068 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn"] Jul 10 19:09:52.758181 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:09:52.758154 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580fe534_ceef_40f1_9d9a_a46fe84457b9.slice/crio-c26de474e86fe5d8fe07d9fabd463af4ec335b50471e3620d4ad22db8c2ac29d WatchSource:0}: Error finding container c26de474e86fe5d8fe07d9fabd463af4ec335b50471e3620d4ad22db8c2ac29d: Status 404 returned error can't find the container with id c26de474e86fe5d8fe07d9fabd463af4ec335b50471e3620d4ad22db8c2ac29d Jul 10 19:09:53.535472 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:53.535436 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" event={"ID":"580fe534-ceef-40f1-9d9a-a46fe84457b9","Type":"ContainerStarted","Data":"c26de474e86fe5d8fe07d9fabd463af4ec335b50471e3620d4ad22db8c2ac29d"} Jul 10 19:09:54.542253 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:54.542218 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" event={"ID":"580fe534-ceef-40f1-9d9a-a46fe84457b9","Type":"ContainerStarted","Data":"a24b129ac22007878477a2065c8ed6ecdb8ea189dd094e3da106afcace0b43de"} Jul 10 19:09:54.562748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:09:54.562699 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" podStartSLOduration=1.452016985 podStartE2EDuration="2.562680445s" totalImagesPullingTime="1.11066346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:09:54.561255322 +0000 UTC m=+571.312160196" watchObservedRunningTime="2026-07-10 19:09:54.562680445 +0000 UTC m=+571.313585320" Jul 10 19:10:02.611043 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:02.611001 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:10:02.611548 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:02.611068 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:10:02.624092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:02.624058 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:10:03.596042 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:03.596014 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-2hcvn" Jul 10 19:10:23.711127 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:23.711087 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:10:23.713160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:23.713138 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:10:23.715832 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:23.715785 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:10:23.717600 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:23.717584 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:10:24.394303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.394270 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs"] Jul 10 19:10:24.398019 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.398000 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.400690 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.400662 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 19:10:24.414258 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.414237 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs"] Jul 10 19:10:24.486181 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.486150 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.486365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.486224 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.486365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.486289 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.486365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.486316 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.486365 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.486335 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtldv\" (UniqueName: \"kubernetes.io/projected/93dd9324-860a-40f3-a341-57572b6fcffa-kube-api-access-jtldv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.486560 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.486384 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93dd9324-860a-40f3-a341-57572b6fcffa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587377 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587335 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587589 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587407 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587589 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587440 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587589 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587458 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587589 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587474 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtldv\" (UniqueName: \"kubernetes.io/projected/93dd9324-860a-40f3-a341-57572b6fcffa-kube-api-access-jtldv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587589 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587501 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93dd9324-860a-40f3-a341-57572b6fcffa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587926 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587899 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587999 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587923 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.587999 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.587956 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.589865 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.589840 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93dd9324-860a-40f3-a341-57572b6fcffa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.590170 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.590150 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93dd9324-860a-40f3-a341-57572b6fcffa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.596474 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.596455 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtldv\" (UniqueName: \"kubernetes.io/projected/93dd9324-860a-40f3-a341-57572b6fcffa-kube-api-access-jtldv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4tgcs\" (UID: \"93dd9324-860a-40f3-a341-57572b6fcffa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.707902 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.707812 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:24.849313 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:24.849292 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs"] Jul 10 19:10:24.851214 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:10:24.851186 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93dd9324_860a_40f3_a341_57572b6fcffa.slice/crio-494a57bcb59235fd482f7ea1071d381ebcdea977449567ef8107ec1549b13bb2 WatchSource:0}: Error finding container 494a57bcb59235fd482f7ea1071d381ebcdea977449567ef8107ec1549b13bb2: Status 404 returned error can't find the container with id 494a57bcb59235fd482f7ea1071d381ebcdea977449567ef8107ec1549b13bb2 Jul 10 19:10:25.694441 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:25.694406 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" event={"ID":"93dd9324-860a-40f3-a341-57572b6fcffa","Type":"ContainerStarted","Data":"8afc3573dd2bc4b074a8d1007a23d8b6f1062a2e7da666aa89a002038950f47f"} Jul 10 19:10:25.694617 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:25.694467 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" event={"ID":"93dd9324-860a-40f3-a341-57572b6fcffa","Type":"ContainerStarted","Data":"494a57bcb59235fd482f7ea1071d381ebcdea977449567ef8107ec1549b13bb2"} Jul 10 19:10:25.714988 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:25.714937 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" podStartSLOduration=1.418917499 podStartE2EDuration="1.714919051s" totalImagesPullingTime="296.001552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:25.712613042 +0000 UTC m=+602.463517919" watchObservedRunningTime="2026-07-10 19:10:25.714919051 +0000 UTC m=+602.465823925" Jul 10 19:10:34.708552 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:34.708517 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:34.713557 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:34.708676 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:34.721357 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:34.721333 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:10:34.749881 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:10:34.749855 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4tgcs" Jul 10 19:12:57.059205 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.059171 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-98f7fc48c-b77sg"] Jul 10 19:12:57.059714 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.059408 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-98f7fc48c-b77sg" podUID="da6e4e8b-40f7-49f4-a82f-e75a35399b64" containerName="manager" containerID="cri-o://25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b" gracePeriod=30 Jul 10 19:12:57.315478 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.315455 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:12:57.404182 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.404153 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6l2h\" (UniqueName: \"kubernetes.io/projected/da6e4e8b-40f7-49f4-a82f-e75a35399b64-kube-api-access-g6l2h\") pod \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\" (UID: \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\") " Jul 10 19:12:57.404182 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.404187 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6e4e8b-40f7-49f4-a82f-e75a35399b64-webhook-cert\") pod \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\" (UID: \"da6e4e8b-40f7-49f4-a82f-e75a35399b64\") " Jul 10 19:12:57.406370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.406341 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6e4e8b-40f7-49f4-a82f-e75a35399b64-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "da6e4e8b-40f7-49f4-a82f-e75a35399b64" (UID: "da6e4e8b-40f7-49f4-a82f-e75a35399b64"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:12:57.406467 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.406438 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6e4e8b-40f7-49f4-a82f-e75a35399b64-kube-api-access-g6l2h" (OuterVolumeSpecName: "kube-api-access-g6l2h") pod "da6e4e8b-40f7-49f4-a82f-e75a35399b64" (UID: "da6e4e8b-40f7-49f4-a82f-e75a35399b64"). InnerVolumeSpecName "kube-api-access-g6l2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:12:57.448222 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.448185 2549 generic.go:356] "Generic (PLEG): container finished" podID="da6e4e8b-40f7-49f4-a82f-e75a35399b64" containerID="25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b" exitCode=0 Jul 10 19:12:57.448389 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.448237 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98f7fc48c-b77sg" Jul 10 19:12:57.448389 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.448266 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98f7fc48c-b77sg" event={"ID":"da6e4e8b-40f7-49f4-a82f-e75a35399b64","Type":"ContainerDied","Data":"25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b"} Jul 10 19:12:57.448389 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.448342 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98f7fc48c-b77sg" event={"ID":"da6e4e8b-40f7-49f4-a82f-e75a35399b64","Type":"ContainerDied","Data":"e0761a3484987bdd8b5f9b267377132a4cfc0c526efb32cb54c5601bd007a315"} Jul 10 19:12:57.448389 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.448367 2549 scope.go:117] "RemoveContainer" containerID="25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b" Jul 10 19:12:57.458287 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.458260 2549 scope.go:117] "RemoveContainer" containerID="25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b" Jul 10 19:12:57.458632 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:12:57.458609 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b\": container with ID starting with 25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b not found: ID does not exist" containerID="25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b" Jul 10 19:12:57.458736 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.458641 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b"} err="failed to get container status \"25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b\": rpc error: code = NotFound desc = could not find container \"25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b\": container with ID starting with 25b4ab715846106133a994ddeb498ebaf34d33b0dbb85f7f45238f29fff4fb7b not found: ID does not exist" Jul 10 19:12:57.471394 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.471369 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-98f7fc48c-b77sg"] Jul 10 19:12:57.476259 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.476235 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-98f7fc48c-b77sg"] Jul 10 19:12:57.505524 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.505495 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6l2h\" (UniqueName: \"kubernetes.io/projected/da6e4e8b-40f7-49f4-a82f-e75a35399b64-kube-api-access-g6l2h\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:12:57.505524 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.505523 2549 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6e4e8b-40f7-49f4-a82f-e75a35399b64-webhook-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:12:57.806228 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:57.806145 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6e4e8b-40f7-49f4-a82f-e75a35399b64" path="/var/lib/kubelet/pods/da6e4e8b-40f7-49f4-a82f-e75a35399b64/volumes" Jul 10 19:12:58.218420 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.218383 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-98f7fc48c-ll6c7"] Jul 10 19:12:58.218820 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.218760 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6e4e8b-40f7-49f4-a82f-e75a35399b64" containerName="manager" Jul 10 19:12:58.218820 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.218773 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6e4e8b-40f7-49f4-a82f-e75a35399b64" containerName="manager" Jul 10 19:12:58.218898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.218849 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6e4e8b-40f7-49f4-a82f-e75a35399b64" containerName="manager" Jul 10 19:12:58.223044 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.223026 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:58.226174 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.226147 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 19:12:58.226292 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.226251 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 19:12:58.227160 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.227144 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-l6jrh\"" Jul 10 19:12:58.242108 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.242083 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-98f7fc48c-ll6c7"] Jul 10 19:12:58.313233 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.313196 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eb2b148-4536-4842-8a49-1a67774f9d04-webhook-cert\") pod \"maas-controller-98f7fc48c-ll6c7\" (UID: \"3eb2b148-4536-4842-8a49-1a67774f9d04\") " pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:58.313410 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.313243 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78jms\" (UniqueName: \"kubernetes.io/projected/3eb2b148-4536-4842-8a49-1a67774f9d04-kube-api-access-78jms\") pod \"maas-controller-98f7fc48c-ll6c7\" (UID: \"3eb2b148-4536-4842-8a49-1a67774f9d04\") " pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:58.413697 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.413657 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eb2b148-4536-4842-8a49-1a67774f9d04-webhook-cert\") pod \"maas-controller-98f7fc48c-ll6c7\" (UID: \"3eb2b148-4536-4842-8a49-1a67774f9d04\") " pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:58.413925 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.413710 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78jms\" (UniqueName: \"kubernetes.io/projected/3eb2b148-4536-4842-8a49-1a67774f9d04-kube-api-access-78jms\") pod \"maas-controller-98f7fc48c-ll6c7\" (UID: \"3eb2b148-4536-4842-8a49-1a67774f9d04\") " pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:58.416380 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.416354 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eb2b148-4536-4842-8a49-1a67774f9d04-webhook-cert\") pod \"maas-controller-98f7fc48c-ll6c7\" (UID: \"3eb2b148-4536-4842-8a49-1a67774f9d04\") " pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:58.423515 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.423485 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78jms\" (UniqueName: \"kubernetes.io/projected/3eb2b148-4536-4842-8a49-1a67774f9d04-kube-api-access-78jms\") pod \"maas-controller-98f7fc48c-ll6c7\" (UID: \"3eb2b148-4536-4842-8a49-1a67774f9d04\") " pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:58.533957 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.533852 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:58.681514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.681480 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-98f7fc48c-ll6c7"] Jul 10 19:12:58.684765 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:12:58.684738 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb2b148_4536_4842_8a49_1a67774f9d04.slice/crio-c4e2707f66100ec6a8c412e4b00b56480e29161a5807591b078501ec8fa9e006 WatchSource:0}: Error finding container c4e2707f66100ec6a8c412e4b00b56480e29161a5807591b078501ec8fa9e006: Status 404 returned error can't find the container with id c4e2707f66100ec6a8c412e4b00b56480e29161a5807591b078501ec8fa9e006 Jul 10 19:12:58.686140 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:58.686122 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:12:59.460591 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:59.460504 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98f7fc48c-ll6c7" event={"ID":"3eb2b148-4536-4842-8a49-1a67774f9d04","Type":"ContainerStarted","Data":"74aa30ab42c4502eb99e7a598c5071606ed2ba084c4f8cc953de4bb54f504f3e"} Jul 10 19:12:59.460591 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:59.460555 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-98f7fc48c-ll6c7" event={"ID":"3eb2b148-4536-4842-8a49-1a67774f9d04","Type":"ContainerStarted","Data":"c4e2707f66100ec6a8c412e4b00b56480e29161a5807591b078501ec8fa9e006"} Jul 10 19:12:59.461033 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:59.460594 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:12:59.483771 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:12:59.483558 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-98f7fc48c-ll6c7" podStartSLOduration=0.976107536 podStartE2EDuration="1.483537026s" totalImagesPullingTime="507.42949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:12:59.483502341 +0000 UTC m=+756.234407228" watchObservedRunningTime="2026-07-10 19:12:59.483537026 +0000 UTC m=+756.234441901" Jul 10 19:13:10.471625 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:13:10.471585 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-98f7fc48c-ll6c7" Jul 10 19:15:23.740930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:15:23.740840 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:15:23.745034 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:15:23.745011 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:15:23.746612 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:15:23.746585 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:15:23.750434 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:15:23.750414 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:17:43.599817 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:43.599755 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc"] Jul 10 19:17:43.602652 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:43.599998 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" podUID="65fe6cad-5bdf-4df8-acdd-404acc8e55aa" containerName="manager" containerID="cri-o://c41c85499880b97a4a5b98ece383e5072bcaed3cd597bda395691c621e00c9e5" gracePeriod=10 Jul 10 19:17:46.794105 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:46.794060 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" podUID="65fe6cad-5bdf-4df8-acdd-404acc8e55aa" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Jul 10 19:17:46.862341 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:46.862299 2549 generic.go:356] "Generic (PLEG): container finished" podID="65fe6cad-5bdf-4df8-acdd-404acc8e55aa" containerID="c41c85499880b97a4a5b98ece383e5072bcaed3cd597bda395691c621e00c9e5" exitCode=0 Jul 10 19:17:46.862498 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:46.862364 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" event={"ID":"65fe6cad-5bdf-4df8-acdd-404acc8e55aa","Type":"ContainerDied","Data":"c41c85499880b97a4a5b98ece383e5072bcaed3cd597bda395691c621e00c9e5"} Jul 10 19:17:46.961136 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:46.961107 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:17:47.062338 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.062250 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzzxh\" (UniqueName: \"kubernetes.io/projected/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-kube-api-access-xzzxh\") pod \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\" (UID: \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\") " Jul 10 19:17:47.062511 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.062361 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-extensions-socket-volume\") pod \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\" (UID: \"65fe6cad-5bdf-4df8-acdd-404acc8e55aa\") " Jul 10 19:17:47.062749 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.062725 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "65fe6cad-5bdf-4df8-acdd-404acc8e55aa" (UID: "65fe6cad-5bdf-4df8-acdd-404acc8e55aa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:17:47.064453 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.064433 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-kube-api-access-xzzxh" (OuterVolumeSpecName: "kube-api-access-xzzxh") pod "65fe6cad-5bdf-4df8-acdd-404acc8e55aa" (UID: "65fe6cad-5bdf-4df8-acdd-404acc8e55aa"). InnerVolumeSpecName "kube-api-access-xzzxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:17:47.163526 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.163484 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzzxh\" (UniqueName: \"kubernetes.io/projected/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-kube-api-access-xzzxh\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:17:47.163526 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.163518 2549 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65fe6cad-5bdf-4df8-acdd-404acc8e55aa-extensions-socket-volume\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:17:47.868485 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.868443 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" event={"ID":"65fe6cad-5bdf-4df8-acdd-404acc8e55aa","Type":"ContainerDied","Data":"3ec4a4312fc7935006a160793b98451782fd867ba6e5b514bb916ca4f4ed193f"} Jul 10 19:17:47.868863 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.868460 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc" Jul 10 19:17:47.868863 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.868538 2549 scope.go:117] "RemoveContainer" containerID="c41c85499880b97a4a5b98ece383e5072bcaed3cd597bda395691c621e00c9e5" Jul 10 19:17:47.886163 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.886117 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc"] Jul 10 19:17:47.889148 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:47.889127 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c8xsc"] Jul 10 19:17:49.804869 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:17:49.804832 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65fe6cad-5bdf-4df8-acdd-404acc8e55aa" path="/var/lib/kubelet/pods/65fe6cad-5bdf-4df8-acdd-404acc8e55aa/volumes" Jul 10 19:18:44.687997 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.687962 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9"] Jul 10 19:18:44.688415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.688354 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65fe6cad-5bdf-4df8-acdd-404acc8e55aa" containerName="manager" Jul 10 19:18:44.688415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.688367 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="65fe6cad-5bdf-4df8-acdd-404acc8e55aa" containerName="manager" Jul 10 19:18:44.688484 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.688431 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="65fe6cad-5bdf-4df8-acdd-404acc8e55aa" containerName="manager" Jul 10 19:18:44.692600 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.692584 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:44.695420 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.695392 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zhzck\"" Jul 10 19:18:44.702925 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.702898 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9"] Jul 10 19:18:44.774525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.774479 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f189776-e68f-4fc6-90b5-e69c4914ea55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c6qh9\" (UID: \"7f189776-e68f-4fc6-90b5-e69c4914ea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:44.774525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.774519 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2mzx\" (UniqueName: \"kubernetes.io/projected/7f189776-e68f-4fc6-90b5-e69c4914ea55-kube-api-access-r2mzx\") pod \"kuadrant-operator-controller-manager-86878c769f-c6qh9\" (UID: \"7f189776-e68f-4fc6-90b5-e69c4914ea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:44.875486 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.875437 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f189776-e68f-4fc6-90b5-e69c4914ea55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c6qh9\" (UID: \"7f189776-e68f-4fc6-90b5-e69c4914ea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:44.875486 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.875486 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2mzx\" (UniqueName: \"kubernetes.io/projected/7f189776-e68f-4fc6-90b5-e69c4914ea55-kube-api-access-r2mzx\") pod \"kuadrant-operator-controller-manager-86878c769f-c6qh9\" (UID: \"7f189776-e68f-4fc6-90b5-e69c4914ea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:44.875910 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.875888 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f189776-e68f-4fc6-90b5-e69c4914ea55-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c6qh9\" (UID: \"7f189776-e68f-4fc6-90b5-e69c4914ea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:44.884291 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:44.884257 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2mzx\" (UniqueName: \"kubernetes.io/projected/7f189776-e68f-4fc6-90b5-e69c4914ea55-kube-api-access-r2mzx\") pod \"kuadrant-operator-controller-manager-86878c769f-c6qh9\" (UID: \"7f189776-e68f-4fc6-90b5-e69c4914ea55\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:45.003688 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:45.003594 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:45.144925 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:45.144895 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9"] Jul 10 19:18:45.146992 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:18:45.146961 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f189776_e68f_4fc6_90b5_e69c4914ea55.slice/crio-7b4efb90e3d81f14a8ac562702a83b4a2bdee87957cb91f69d414d6a14162a15 WatchSource:0}: Error finding container 7b4efb90e3d81f14a8ac562702a83b4a2bdee87957cb91f69d414d6a14162a15: Status 404 returned error can't find the container with id 7b4efb90e3d81f14a8ac562702a83b4a2bdee87957cb91f69d414d6a14162a15 Jul 10 19:18:46.152259 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:46.152223 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" event={"ID":"7f189776-e68f-4fc6-90b5-e69c4914ea55","Type":"ContainerStarted","Data":"1b4d22a4f8f0635a8d59e6033434468be87807f8fee15c7a7c74fa25e443f461"} Jul 10 19:18:46.152651 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:46.152277 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" event={"ID":"7f189776-e68f-4fc6-90b5-e69c4914ea55","Type":"ContainerStarted","Data":"7b4efb90e3d81f14a8ac562702a83b4a2bdee87957cb91f69d414d6a14162a15"} Jul 10 19:18:46.152651 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:46.152350 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:18:46.188701 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:46.188644 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" podStartSLOduration=2.188627834 podStartE2EDuration="2.188627834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:18:46.18725662 +0000 UTC m=+1102.938161518" watchObservedRunningTime="2026-07-10 19:18:46.188627834 +0000 UTC m=+1102.939532709" Jul 10 19:18:57.159173 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:18:57.159094 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c6qh9" Jul 10 19:20:23.771027 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:20:23.770993 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:20:23.776019 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:20:23.775989 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:20:23.778166 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:20:23.778146 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:20:23.782759 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:20:23.782741 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:25:12.579342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.579308 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf"] Jul 10 19:25:12.583643 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.583620 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.586307 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.586281 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5d271552-openshift-default-dockercfg-qtzbq\"" Jul 10 19:25:12.606420 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.606389 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf"] Jul 10 19:25:12.707319 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707279 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-credential-socket\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.707492 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707327 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-podinfo\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.707492 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707398 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdpgk\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-kube-api-access-sdpgk\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.707492 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707431 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-token\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.707492 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707488 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-envoy\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.707635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707541 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-socket\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.707635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707598 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-certs\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.707635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707626 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-data\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.707727 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.707659 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istiod-ca-cert\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808665 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808626 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-credential-socket\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808665 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808671 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-podinfo\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808703 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdpgk\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-kube-api-access-sdpgk\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808721 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-token\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808746 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-envoy\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808783 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-socket\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808850 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-certs\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808882 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-data\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.808980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.808919 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istiod-ca-cert\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.809342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.809140 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-credential-socket\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.809342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.809237 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-socket\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.809342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.809277 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-data\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.809478 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.809422 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-certs\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.809862 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.809842 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istiod-ca-cert\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.811262 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.811240 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-envoy\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.811520 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.811491 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-podinfo\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.816684 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.816660 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-token\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.816924 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.816904 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdpgk\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-kube-api-access-sdpgk\") pod \"e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:12.894665 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:12.894564 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:13.039134 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:13.039110 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf"] Jul 10 19:25:13.045707 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:13.045664 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:13.045829 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:13.045754 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:13.045829 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:13.045816 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:14.039029 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.038983 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" event={"ID":"7bcb997c-b79e-4688-acce-c0e427d1e8b6","Type":"ContainerStarted","Data":"5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15"} Jul 10 19:25:14.039451 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.039052 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" event={"ID":"7bcb997c-b79e-4688-acce-c0e427d1e8b6","Type":"ContainerStarted","Data":"924c945491f930afeb8582d1a6eaab262fc73811b67109611a13aebc95b381a3"} Jul 10 19:25:14.069265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.069221 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb"] Jul 10 19:25:14.074183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.074155 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.076837 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.076783 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5d271552-serving-cert\"" Jul 10 19:25:14.083127 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.083081 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" podStartSLOduration=2.08306482 podStartE2EDuration="2.08306482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:14.081549279 +0000 UTC m=+1490.832454179" watchObservedRunningTime="2026-07-10 19:25:14.08306482 +0000 UTC m=+1490.833969693" Jul 10 19:25:14.089035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.089004 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb"] Jul 10 19:25:14.222239 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.222196 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-openshift-service-ca\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.222421 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.222249 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-maas-api-tls\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.222421 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.222379 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ld47\" (UniqueName: \"kubernetes.io/projected/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-kube-api-access-5ld47\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.323041 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.323006 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ld47\" (UniqueName: \"kubernetes.io/projected/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-kube-api-access-5ld47\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.323232 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.323067 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-openshift-service-ca\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.323232 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.323094 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-maas-api-tls\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.323778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.323751 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-openshift-service-ca\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.325768 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.325750 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-maas-api-tls\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.331138 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.331107 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ld47\" (UniqueName: \"kubernetes.io/projected/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-kube-api-access-5ld47\") pod \"maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.385692 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.385650 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:14.533652 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.533627 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb"] Jul 10 19:25:14.537070 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:25:14.537031 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8404a65c_bde5_4ce6_8efd_b88c9bb80c72.slice/crio-dbb8ad3959426e5972b747fe466fda0e4e9da320845a9eb379d47b5befaab95e WatchSource:0}: Error finding container dbb8ad3959426e5972b747fe466fda0e4e9da320845a9eb379d47b5befaab95e: Status 404 returned error can't find the container with id dbb8ad3959426e5972b747fe466fda0e4e9da320845a9eb379d47b5befaab95e Jul 10 19:25:14.538499 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.538479 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:25:14.894956 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.894921 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:14.900125 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:14.900097 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:15.046488 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:15.046451 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" event={"ID":"8404a65c-bde5-4ce6-8efd-b88c9bb80c72","Type":"ContainerStarted","Data":"1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be"} Jul 10 19:25:15.046998 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:15.046519 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" event={"ID":"8404a65c-bde5-4ce6-8efd-b88c9bb80c72","Type":"ContainerStarted","Data":"dbb8ad3959426e5972b747fe466fda0e4e9da320845a9eb379d47b5befaab95e"} Jul 10 19:25:15.046998 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:15.046677 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:15.046998 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:15.046721 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:15.047893 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:15.047875 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:15.066846 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:15.066774 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" podStartSLOduration=0.642115179 podStartE2EDuration="1.06675652s" totalImagesPullingTime="424.641341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:15.065326585 +0000 UTC m=+1491.816231460" watchObservedRunningTime="2026-07-10 19:25:15.06675652 +0000 UTC m=+1491.817661394" Jul 10 19:25:19.415934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.415895 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf"] Jul 10 19:25:19.416419 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.416111 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" podUID="7bcb997c-b79e-4688-acce-c0e427d1e8b6" containerName="istio-proxy" containerID="cri-o://5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15" gracePeriod=30 Jul 10 19:25:19.641212 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.641181 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb"] Jul 10 19:25:19.641438 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.641402 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" podUID="8404a65c-bde5-4ce6-8efd-b88c9bb80c72" containerName="maas-api" containerID="cri-o://1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be" gracePeriod=30 Jul 10 19:25:19.897227 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.897204 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:19.976632 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.976601 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-openshift-service-ca\") pod \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " Jul 10 19:25:19.976821 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.976662 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-maas-api-tls\") pod \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " Jul 10 19:25:19.976821 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.976695 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ld47\" (UniqueName: \"kubernetes.io/projected/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-kube-api-access-5ld47\") pod \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\" (UID: \"8404a65c-bde5-4ce6-8efd-b88c9bb80c72\") " Jul 10 19:25:19.977018 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.976995 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8404a65c-bde5-4ce6-8efd-b88c9bb80c72" (UID: "8404a65c-bde5-4ce6-8efd-b88c9bb80c72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:25:19.979250 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.979217 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8404a65c-bde5-4ce6-8efd-b88c9bb80c72" (UID: "8404a65c-bde5-4ce6-8efd-b88c9bb80c72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:25:19.979359 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:19.979284 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-kube-api-access-5ld47" (OuterVolumeSpecName: "kube-api-access-5ld47") pod "8404a65c-bde5-4ce6-8efd-b88c9bb80c72" (UID: "8404a65c-bde5-4ce6-8efd-b88c9bb80c72"). InnerVolumeSpecName "kube-api-access-5ld47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:20.076134 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.076098 2549 generic.go:356] "Generic (PLEG): container finished" podID="8404a65c-bde5-4ce6-8efd-b88c9bb80c72" containerID="1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be" exitCode=0 Jul 10 19:25:20.076303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.076151 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" Jul 10 19:25:20.076303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.076185 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" event={"ID":"8404a65c-bde5-4ce6-8efd-b88c9bb80c72","Type":"ContainerDied","Data":"1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be"} Jul 10 19:25:20.076303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.076244 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb" event={"ID":"8404a65c-bde5-4ce6-8efd-b88c9bb80c72","Type":"ContainerDied","Data":"dbb8ad3959426e5972b747fe466fda0e4e9da320845a9eb379d47b5befaab95e"} Jul 10 19:25:20.076303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.076267 2549 scope.go:117] "RemoveContainer" containerID="1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be" Jul 10 19:25:20.077430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.077412 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-maas-api-tls\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:20.077535 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.077434 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ld47\" (UniqueName: \"kubernetes.io/projected/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-kube-api-access-5ld47\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:20.077535 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.077447 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8404a65c-bde5-4ce6-8efd-b88c9bb80c72-openshift-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:20.085776 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.085761 2549 scope.go:117] "RemoveContainer" containerID="1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be" Jul 10 19:25:20.086047 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:25:20.086031 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be\": container with ID starting with 1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be not found: ID does not exist" containerID="1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be" Jul 10 19:25:20.086103 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.086055 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be"} err="failed to get container status \"1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be\": rpc error: code = NotFound desc = could not find container \"1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be\": container with ID starting with 1ce8e9845a655b1567e9bc90e4b65a051a38e86a9fce6eacf2b93ca990cb27be not found: ID does not exist" Jul 10 19:25:20.099200 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.099170 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb"] Jul 10 19:25:20.108780 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:20.108758 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5d271552-84f56b9d45-mbrnb"] Jul 10 19:25:21.804279 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:21.804239 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8404a65c-bde5-4ce6-8efd-b88c9bb80c72" path="/var/lib/kubelet/pods/8404a65c-bde5-4ce6-8efd-b88c9bb80c72/volumes" Jul 10 19:25:23.807355 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:23.807322 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:25:23.812352 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:23.812325 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:25:23.813871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:23.813852 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:25:23.818706 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:23.818689 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:25:24.679599 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.679538 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:24.822379 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822351 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-envoy\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.822781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822387 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istiod-ca-cert\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.822781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822436 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-credential-socket\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.822781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822478 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-socket\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.822781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822509 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-podinfo\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.822781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822540 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-token\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.822781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822582 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-data\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.822781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822622 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-certs\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.822781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822656 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdpgk\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-kube-api-access-sdpgk\") pod \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\" (UID: \"7bcb997c-b79e-4688-acce-c0e427d1e8b6\") " Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822784 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822832 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822919 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822924 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.822991 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.823007 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.823010 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.823035 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:24.823206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.823048 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:24.825021 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.824991 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:24.825173 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.825127 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:25:24.825446 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.825430 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-kube-api-access-sdpgk" (OuterVolumeSpecName: "kube-api-access-sdpgk") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "kube-api-access-sdpgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:24.825495 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.825449 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bcb997c-b79e-4688-acce-c0e427d1e8b6" (UID: "7bcb997c-b79e-4688-acce-c0e427d1e8b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:24.924057 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.924019 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:24.924057 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.924050 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:24.924057 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.924058 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:24.924057 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.924066 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcb997c-b79e-4688-acce-c0e427d1e8b6-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:24.924321 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:24.924077 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdpgk\" (UniqueName: \"kubernetes.io/projected/7bcb997c-b79e-4688-acce-c0e427d1e8b6-kube-api-access-sdpgk\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:25.105148 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.105114 2549 generic.go:356] "Generic (PLEG): container finished" podID="7bcb997c-b79e-4688-acce-c0e427d1e8b6" containerID="5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15" exitCode=0 Jul 10 19:25:25.105325 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.105165 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" Jul 10 19:25:25.105325 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.105205 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" event={"ID":"7bcb997c-b79e-4688-acce-c0e427d1e8b6","Type":"ContainerDied","Data":"5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15"} Jul 10 19:25:25.105325 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.105262 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf" event={"ID":"7bcb997c-b79e-4688-acce-c0e427d1e8b6","Type":"ContainerDied","Data":"924c945491f930afeb8582d1a6eaab262fc73811b67109611a13aebc95b381a3"} Jul 10 19:25:25.105325 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.105286 2549 scope.go:117] "RemoveContainer" containerID="5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15" Jul 10 19:25:25.114697 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.114682 2549 scope.go:117] "RemoveContainer" containerID="5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15" Jul 10 19:25:25.114973 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:25:25.114952 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15\": container with ID starting with 5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15 not found: ID does not exist" containerID="5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15" Jul 10 19:25:25.115042 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.114979 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15"} err="failed to get container status \"5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15\": rpc error: code = NotFound desc = could not find container \"5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15\": container with ID starting with 5d22b56c3d2425343efdd45b4916d85c6a90c4d36453a5e810457a6bf8922d15 not found: ID does not exist" Jul 10 19:25:25.127423 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.127401 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf"] Jul 10 19:25:25.132678 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.132654 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5d271552-openshift-default-85bcf5599c-9jvtf"] Jul 10 19:25:25.805767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:25.805732 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bcb997c-b79e-4688-acce-c0e427d1e8b6" path="/var/lib/kubelet/pods/7bcb997c-b79e-4688-acce-c0e427d1e8b6/volumes" Jul 10 19:25:26.046014 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.045975 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn"] Jul 10 19:25:26.046424 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.046405 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8404a65c-bde5-4ce6-8efd-b88c9bb80c72" containerName="maas-api" Jul 10 19:25:26.046517 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.046454 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8404a65c-bde5-4ce6-8efd-b88c9bb80c72" containerName="maas-api" Jul 10 19:25:26.046517 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.046475 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bcb997c-b79e-4688-acce-c0e427d1e8b6" containerName="istio-proxy" Jul 10 19:25:26.046517 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.046483 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bcb997c-b79e-4688-acce-c0e427d1e8b6" containerName="istio-proxy" Jul 10 19:25:26.046719 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.046646 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bcb997c-b79e-4688-acce-c0e427d1e8b6" containerName="istio-proxy" Jul 10 19:25:26.046719 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.046660 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8404a65c-bde5-4ce6-8efd-b88c9bb80c72" containerName="maas-api" Jul 10 19:25:26.050240 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.050216 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.052840 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.052820 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b26d5934-gw-openshift-default-dockercfg-d6v7s\"" Jul 10 19:25:26.063237 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.063213 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn"] Jul 10 19:25:26.135079 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135043 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-istio-token\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.135079 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135084 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-envoy\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.135323 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135099 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-data\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.135323 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135243 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-socket\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.135323 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135283 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-certs\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.135323 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135314 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chs55\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-kube-api-access-chs55\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.135502 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135365 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a390d19-d7fa-45a8-b0b3-354661943753-istiod-ca-cert\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.135502 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135412 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-credential-socket\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.135502 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.135437 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a390d19-d7fa-45a8-b0b3-354661943753-istio-podinfo\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236145 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236107 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-credential-socket\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236145 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236144 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a390d19-d7fa-45a8-b0b3-354661943753-istio-podinfo\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236192 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-istio-token\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236212 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-envoy\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236230 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-data\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236293 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-socket\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236325 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-certs\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236370 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chs55\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-kube-api-access-chs55\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236615 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-data\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236708 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-certs\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236728 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-socket\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.236871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.236747 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a390d19-d7fa-45a8-b0b3-354661943753-istiod-ca-cert\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.237476 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.237333 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a390d19-d7fa-45a8-b0b3-354661943753-istiod-ca-cert\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.237537 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.237483 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-credential-socket\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.238819 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.238766 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-envoy\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.239023 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.239000 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a390d19-d7fa-45a8-b0b3-354661943753-istio-podinfo\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.244060 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.244039 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-istio-token\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.244351 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.244332 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chs55\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-kube-api-access-chs55\") pod \"e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.363371 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.363324 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:26.508118 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.508092 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn"] Jul 10 19:25:26.511040 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:25:26.511015 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a390d19_d7fa_45a8_b0b3_354661943753.slice/crio-d75d292110a441799e8ea5af7935e4190f94d6625cee95519a37c9276cf57d28 WatchSource:0}: Error finding container d75d292110a441799e8ea5af7935e4190f94d6625cee95519a37c9276cf57d28: Status 404 returned error can't find the container with id d75d292110a441799e8ea5af7935e4190f94d6625cee95519a37c9276cf57d28 Jul 10 19:25:26.513576 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.513540 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:26.513667 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.513606 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:26.513667 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:26.513634 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:27.117784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.117744 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" event={"ID":"2a390d19-d7fa-45a8-b0b3-354661943753","Type":"ContainerStarted","Data":"bbc377a30dc57ddf5f4a440f550ca1e85b410b73ad746f8f384623ca8ba77bfc"} Jul 10 19:25:27.118283 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.117842 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" event={"ID":"2a390d19-d7fa-45a8-b0b3-354661943753","Type":"ContainerStarted","Data":"d75d292110a441799e8ea5af7935e4190f94d6625cee95519a37c9276cf57d28"} Jul 10 19:25:27.140876 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.140820 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" podStartSLOduration=1.140781314 podStartE2EDuration="1.140781314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:27.138320196 +0000 UTC m=+1503.889225112" watchObservedRunningTime="2026-07-10 19:25:27.140781314 +0000 UTC m=+1503.891686187" Jul 10 19:25:27.363949 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.363913 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:27.369208 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.369137 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:27.479053 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.479021 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5"] Jul 10 19:25:27.482666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.482645 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:27.485120 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.485099 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b26d5934-serving-cert\"" Jul 10 19:25:27.494149 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.494124 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5"] Jul 10 19:25:27.650985 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.650896 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc728\" (UniqueName: \"kubernetes.io/projected/864efdde-649d-427e-a41d-ab18a1e1306c-kube-api-access-lc728\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:27.650985 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.650966 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:27.651194 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.651026 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864efdde-649d-427e-a41d-ab18a1e1306c-openshift-service-ca\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:27.751522 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.751478 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864efdde-649d-427e-a41d-ab18a1e1306c-openshift-service-ca\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:27.751702 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.751569 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc728\" (UniqueName: \"kubernetes.io/projected/864efdde-649d-427e-a41d-ab18a1e1306c-kube-api-access-lc728\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:27.751702 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.751625 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:27.751776 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:25:27.751724 2549 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-serving-cert: secret "maas-api-e2e-migrate-b26d5934-serving-cert" not found Jul 10 19:25:27.751840 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:25:27.751817 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls podName:864efdde-649d-427e-a41d-ab18a1e1306c nodeName:}" failed. No retries permitted until 2026-07-10 19:25:28.251768278 +0000 UTC m=+1505.002673131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls") pod "maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" (UID: "864efdde-649d-427e-a41d-ab18a1e1306c") : secret "maas-api-e2e-migrate-b26d5934-serving-cert" not found Jul 10 19:25:27.752188 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.752157 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864efdde-649d-427e-a41d-ab18a1e1306c-openshift-service-ca\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:27.761829 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:27.761805 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc728\" (UniqueName: \"kubernetes.io/projected/864efdde-649d-427e-a41d-ab18a1e1306c-kube-api-access-lc728\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:28.124663 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:28.124623 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:28.125784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:28.125766 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:28.257219 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:28.257184 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:28.259808 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:28.259768 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls\") pod \"maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:28.396571 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:28.396476 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:28.538877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:28.538843 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5"] Jul 10 19:25:28.543500 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:25:28.543459 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864efdde_649d_427e_a41d_ab18a1e1306c.slice/crio-31c090769e7752efd73ac14605d87022827127211bcb79908029d43d7ec77174 WatchSource:0}: Error finding container 31c090769e7752efd73ac14605d87022827127211bcb79908029d43d7ec77174: Status 404 returned error can't find the container with id 31c090769e7752efd73ac14605d87022827127211bcb79908029d43d7ec77174 Jul 10 19:25:29.131213 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:29.131172 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" event={"ID":"864efdde-649d-427e-a41d-ab18a1e1306c","Type":"ContainerStarted","Data":"78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66"} Jul 10 19:25:29.131635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:29.131245 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:29.131635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:29.131273 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" event={"ID":"864efdde-649d-427e-a41d-ab18a1e1306c","Type":"ContainerStarted","Data":"31c090769e7752efd73ac14605d87022827127211bcb79908029d43d7ec77174"} Jul 10 19:25:29.160546 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:29.160444 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" podStartSLOduration=1.796014658 podStartE2EDuration="2.160420523s" totalImagesPullingTime="364.405865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:29.158250044 +0000 UTC m=+1505.909154921" watchObservedRunningTime="2026-07-10 19:25:29.160420523 +0000 UTC m=+1505.911325400" Jul 10 19:25:31.982877 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:31.982832 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn"] Jul 10 19:25:31.983437 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:31.983146 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" podUID="2a390d19-d7fa-45a8-b0b3-354661943753" containerName="istio-proxy" containerID="cri-o://bbc377a30dc57ddf5f4a440f550ca1e85b410b73ad746f8f384623ca8ba77bfc" gracePeriod=30 Jul 10 19:25:32.197852 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.197815 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5"] Jul 10 19:25:32.198082 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.198056 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" podUID="864efdde-649d-427e-a41d-ab18a1e1306c" containerName="maas-api" containerID="cri-o://78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66" gracePeriod=30 Jul 10 19:25:32.451029 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.451004 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:32.595746 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.595705 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864efdde-649d-427e-a41d-ab18a1e1306c-openshift-service-ca\") pod \"864efdde-649d-427e-a41d-ab18a1e1306c\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " Jul 10 19:25:32.595940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.595845 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc728\" (UniqueName: \"kubernetes.io/projected/864efdde-649d-427e-a41d-ab18a1e1306c-kube-api-access-lc728\") pod \"864efdde-649d-427e-a41d-ab18a1e1306c\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " Jul 10 19:25:32.595940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.595901 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls\") pod \"864efdde-649d-427e-a41d-ab18a1e1306c\" (UID: \"864efdde-649d-427e-a41d-ab18a1e1306c\") " Jul 10 19:25:32.596148 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.596116 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864efdde-649d-427e-a41d-ab18a1e1306c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "864efdde-649d-427e-a41d-ab18a1e1306c" (UID: "864efdde-649d-427e-a41d-ab18a1e1306c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:25:32.598037 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.598004 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "864efdde-649d-427e-a41d-ab18a1e1306c" (UID: "864efdde-649d-427e-a41d-ab18a1e1306c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:25:32.598125 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.598090 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864efdde-649d-427e-a41d-ab18a1e1306c-kube-api-access-lc728" (OuterVolumeSpecName: "kube-api-access-lc728") pod "864efdde-649d-427e-a41d-ab18a1e1306c" (UID: "864efdde-649d-427e-a41d-ab18a1e1306c"). InnerVolumeSpecName "kube-api-access-lc728". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:32.696768 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.696730 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc728\" (UniqueName: \"kubernetes.io/projected/864efdde-649d-427e-a41d-ab18a1e1306c-kube-api-access-lc728\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:32.696768 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.696760 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864efdde-649d-427e-a41d-ab18a1e1306c-maas-api-tls\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:32.696768 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:32.696770 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864efdde-649d-427e-a41d-ab18a1e1306c-openshift-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:33.154871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.154837 2549 generic.go:356] "Generic (PLEG): container finished" podID="864efdde-649d-427e-a41d-ab18a1e1306c" containerID="78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66" exitCode=0 Jul 10 19:25:33.155342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.154880 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" Jul 10 19:25:33.155342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.154894 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" event={"ID":"864efdde-649d-427e-a41d-ab18a1e1306c","Type":"ContainerDied","Data":"78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66"} Jul 10 19:25:33.155342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.154948 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5" event={"ID":"864efdde-649d-427e-a41d-ab18a1e1306c","Type":"ContainerDied","Data":"31c090769e7752efd73ac14605d87022827127211bcb79908029d43d7ec77174"} Jul 10 19:25:33.155342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.154979 2549 scope.go:117] "RemoveContainer" containerID="78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66" Jul 10 19:25:33.165398 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.165381 2549 scope.go:117] "RemoveContainer" containerID="78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66" Jul 10 19:25:33.165702 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:25:33.165685 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66\": container with ID starting with 78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66 not found: ID does not exist" containerID="78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66" Jul 10 19:25:33.165740 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.165713 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66"} err="failed to get container status \"78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66\": rpc error: code = NotFound desc = could not find container \"78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66\": container with ID starting with 78cca66798a7b5fe5e077d0ffd096d19f6f9d61972a0a903287d79d06e722b66 not found: ID does not exist" Jul 10 19:25:33.179073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.179047 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5"] Jul 10 19:25:33.182638 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.182614 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b26d5934-86cc87858d-s82m5"] Jul 10 19:25:33.805075 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:33.805036 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="864efdde-649d-427e-a41d-ab18a1e1306c" path="/var/lib/kubelet/pods/864efdde-649d-427e-a41d-ab18a1e1306c/volumes" Jul 10 19:25:37.180222 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.180190 2549 generic.go:356] "Generic (PLEG): container finished" podID="2a390d19-d7fa-45a8-b0b3-354661943753" containerID="bbc377a30dc57ddf5f4a440f550ca1e85b410b73ad746f8f384623ca8ba77bfc" exitCode=0 Jul 10 19:25:37.180635 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.180244 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" event={"ID":"2a390d19-d7fa-45a8-b0b3-354661943753","Type":"ContainerDied","Data":"bbc377a30dc57ddf5f4a440f550ca1e85b410b73ad746f8f384623ca8ba77bfc"} Jul 10 19:25:37.246980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.246920 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:37.440658 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.440621 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-istio-token\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.440850 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.440683 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-envoy\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.440850 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.440706 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-socket\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.440850 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.440740 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-data\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.440850 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.440783 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chs55\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-kube-api-access-chs55\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.440850 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.440839 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-certs\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.441105 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.440978 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a390d19-d7fa-45a8-b0b3-354661943753-istio-podinfo\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.441105 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441027 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-credential-socket\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.441105 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441051 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:37.441105 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441066 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a390d19-d7fa-45a8-b0b3-354661943753-istiod-ca-cert\") pod \"2a390d19-d7fa-45a8-b0b3-354661943753\" (UID: \"2a390d19-d7fa-45a8-b0b3-354661943753\") " Jul 10 19:25:37.441318 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441129 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:37.441318 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441198 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:37.441428 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441327 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:37.441428 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441377 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a390d19-d7fa-45a8-b0b3-354661943753-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:25:37.441665 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441637 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a390d19-d7fa-45a8-b0b3-354661943753-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.441784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441669 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.441784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441684 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.441784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441699 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.441784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.441712 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.443086 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.443061 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:37.443283 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.443260 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a390d19-d7fa-45a8-b0b3-354661943753-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:25:37.443533 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.443514 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:37.443608 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.443527 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-kube-api-access-chs55" (OuterVolumeSpecName: "kube-api-access-chs55") pod "2a390d19-d7fa-45a8-b0b3-354661943753" (UID: "2a390d19-d7fa-45a8-b0b3-354661943753"). InnerVolumeSpecName "kube-api-access-chs55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:37.542348 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.542251 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a390d19-d7fa-45a8-b0b3-354661943753-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.542348 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.542287 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chs55\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-kube-api-access-chs55\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.542348 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.542300 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a390d19-d7fa-45a8-b0b3-354661943753-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:37.542348 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:37.542311 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a390d19-d7fa-45a8-b0b3-354661943753-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:38.185915 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.185887 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" Jul 10 19:25:38.185915 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.185907 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn" event={"ID":"2a390d19-d7fa-45a8-b0b3-354661943753","Type":"ContainerDied","Data":"d75d292110a441799e8ea5af7935e4190f94d6625cee95519a37c9276cf57d28"} Jul 10 19:25:38.186481 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.185974 2549 scope.go:117] "RemoveContainer" containerID="bbc377a30dc57ddf5f4a440f550ca1e85b410b73ad746f8f384623ca8ba77bfc" Jul 10 19:25:38.203897 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.203872 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn"] Jul 10 19:25:38.207570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.207547 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b26d5934-gw-openshift-default-7975d9f875-qb6nn"] Jul 10 19:25:38.291644 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.291614 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb"] Jul 10 19:25:38.292080 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.292063 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a390d19-d7fa-45a8-b0b3-354661943753" containerName="istio-proxy" Jul 10 19:25:38.292080 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.292079 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a390d19-d7fa-45a8-b0b3-354661943753" containerName="istio-proxy" Jul 10 19:25:38.292250 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.292091 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="864efdde-649d-427e-a41d-ab18a1e1306c" containerName="maas-api" Jul 10 19:25:38.292250 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.292097 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="864efdde-649d-427e-a41d-ab18a1e1306c" containerName="maas-api" Jul 10 19:25:38.292250 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.292167 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a390d19-d7fa-45a8-b0b3-354661943753" containerName="istio-proxy" Jul 10 19:25:38.292250 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.292178 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="864efdde-649d-427e-a41d-ab18a1e1306c" containerName="maas-api" Jul 10 19:25:38.296875 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.296850 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.299256 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.299233 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c05b0410-openshift-default-dockercfg-9824n\"" Jul 10 19:25:38.310517 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.310490 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb"] Jul 10 19:25:38.449669 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449572 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-podinfo\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.449669 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449626 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-socket\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.449669 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449648 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-credential-socket\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.449943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449674 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-data\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.449943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449740 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-certs\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.449943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449786 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-envoy\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.449943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449828 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b454ba-b5df-42d0-a41e-eba92ce609b6-istiod-ca-cert\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.449943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449871 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-token\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.449943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.449895 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthrf\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-kube-api-access-mthrf\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.550846 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.550778 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-envoy\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.550846 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.550841 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b454ba-b5df-42d0-a41e-eba92ce609b6-istiod-ca-cert\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.550877 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-token\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.550899 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthrf\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-kube-api-access-mthrf\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.550956 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-podinfo\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551017 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-socket\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551038 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-credential-socket\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551061 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-data\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551091 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-certs\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551467 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551426 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-credential-socket\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551542 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551473 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-socket\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551636 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551611 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-certs\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.551970 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551947 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b454ba-b5df-42d0-a41e-eba92ce609b6-istiod-ca-cert\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.552099 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.551627 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-data\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.553335 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.553313 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-envoy\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.553534 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.553513 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-podinfo\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.561754 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.561733 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-token\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.561906 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.561856 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthrf\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-kube-api-access-mthrf\") pod \"e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.609949 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.609909 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:38.778378 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.778353 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb"] Jul 10 19:25:38.781104 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:25:38.781073 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b454ba_b5df_42d0_a41e_eba92ce609b6.slice/crio-6a1c49c3908504dcf60440a0fbf1b08a0fded95488b51200cb8367aa4aed2cf1 WatchSource:0}: Error finding container 6a1c49c3908504dcf60440a0fbf1b08a0fded95488b51200cb8367aa4aed2cf1: Status 404 returned error can't find the container with id 6a1c49c3908504dcf60440a0fbf1b08a0fded95488b51200cb8367aa4aed2cf1 Jul 10 19:25:38.783342 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.783310 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:38.783407 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.783380 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:38.783452 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:38.783409 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:39.192934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.192902 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" event={"ID":"91b454ba-b5df-42d0-a41e-eba92ce609b6","Type":"ContainerStarted","Data":"de9a464c054fb7c8cc7185d67a2d3b7dcdb01c8c2fc87d5d82fbe2ce2a66795d"} Jul 10 19:25:39.193324 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.192949 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" event={"ID":"91b454ba-b5df-42d0-a41e-eba92ce609b6","Type":"ContainerStarted","Data":"6a1c49c3908504dcf60440a0fbf1b08a0fded95488b51200cb8367aa4aed2cf1"} Jul 10 19:25:39.220898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.220830 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" podStartSLOduration=1.220811094 podStartE2EDuration="1.220811094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:39.217355991 +0000 UTC m=+1515.968260868" watchObservedRunningTime="2026-07-10 19:25:39.220811094 +0000 UTC m=+1515.971715958" Jul 10 19:25:39.563073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.562987 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4"] Jul 10 19:25:39.566720 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.566688 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.569338 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.569311 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c05b0410-serving-cert\"" Jul 10 19:25:39.578557 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.578518 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4"] Jul 10 19:25:39.610904 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.610863 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:39.616398 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.616366 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:39.659992 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.659948 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95a108a-eead-4733-81a3-132be8780409-maas-api-tls\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.660203 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.660061 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95a108a-eead-4733-81a3-132be8780409-openshift-service-ca\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.660203 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.660100 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxs74\" (UniqueName: \"kubernetes.io/projected/f95a108a-eead-4733-81a3-132be8780409-kube-api-access-qxs74\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.761429 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.761387 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95a108a-eead-4733-81a3-132be8780409-maas-api-tls\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.761613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.761491 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95a108a-eead-4733-81a3-132be8780409-openshift-service-ca\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.761613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.761525 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxs74\" (UniqueName: \"kubernetes.io/projected/f95a108a-eead-4733-81a3-132be8780409-kube-api-access-qxs74\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.762267 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.762196 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95a108a-eead-4733-81a3-132be8780409-openshift-service-ca\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.764170 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.764144 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95a108a-eead-4733-81a3-132be8780409-maas-api-tls\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.770301 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.770275 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxs74\" (UniqueName: \"kubernetes.io/projected/f95a108a-eead-4733-81a3-132be8780409-kube-api-access-qxs74\") pod \"maas-api-e2e-ait-c05b0410-7f99669999-9dvm4\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:39.805360 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.805321 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a390d19-d7fa-45a8-b0b3-354661943753" path="/var/lib/kubelet/pods/2a390d19-d7fa-45a8-b0b3-354661943753/volumes" Jul 10 19:25:39.878851 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:39.878787 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:40.021980 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:40.021950 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4"] Jul 10 19:25:40.024464 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:25:40.024435 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95a108a_eead_4733_81a3_132be8780409.slice/crio-360ad98d8717300d8ff061e3121cc8b21293aa2b00a96bd3a348abeb11d6660f WatchSource:0}: Error finding container 360ad98d8717300d8ff061e3121cc8b21293aa2b00a96bd3a348abeb11d6660f: Status 404 returned error can't find the container with id 360ad98d8717300d8ff061e3121cc8b21293aa2b00a96bd3a348abeb11d6660f Jul 10 19:25:40.199066 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:40.198968 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" event={"ID":"f95a108a-eead-4733-81a3-132be8780409","Type":"ContainerStarted","Data":"360ad98d8717300d8ff061e3121cc8b21293aa2b00a96bd3a348abeb11d6660f"} Jul 10 19:25:40.199418 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:40.199274 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:40.200314 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:40.200294 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:41.205596 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:41.205559 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" event={"ID":"f95a108a-eead-4733-81a3-132be8780409","Type":"ContainerStarted","Data":"b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325"} Jul 10 19:25:41.206024 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:41.205934 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:41.224733 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:41.224681 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" podStartSLOduration=1.8226331409999998 podStartE2EDuration="2.224663782s" totalImagesPullingTime="402.030641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:41.223464197 +0000 UTC m=+1517.974369104" watchObservedRunningTime="2026-07-10 19:25:41.224663782 +0000 UTC m=+1517.975568656" Jul 10 19:25:44.919055 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:44.918976 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4"] Jul 10 19:25:44.919480 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:44.919197 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" podUID="f95a108a-eead-4733-81a3-132be8780409" containerName="maas-api" containerID="cri-o://b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325" gracePeriod=30 Jul 10 19:25:45.214059 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.214028 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:45.226233 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.226201 2549 generic.go:356] "Generic (PLEG): container finished" podID="f95a108a-eead-4733-81a3-132be8780409" containerID="b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325" exitCode=0 Jul 10 19:25:45.226375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.226243 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" Jul 10 19:25:45.226375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.226252 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" event={"ID":"f95a108a-eead-4733-81a3-132be8780409","Type":"ContainerDied","Data":"b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325"} Jul 10 19:25:45.226375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.226316 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4" event={"ID":"f95a108a-eead-4733-81a3-132be8780409","Type":"ContainerDied","Data":"360ad98d8717300d8ff061e3121cc8b21293aa2b00a96bd3a348abeb11d6660f"} Jul 10 19:25:45.226375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.226350 2549 scope.go:117] "RemoveContainer" containerID="b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325" Jul 10 19:25:45.262647 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.262368 2549 scope.go:117] "RemoveContainer" containerID="b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325" Jul 10 19:25:45.263043 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:25:45.262952 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325\": container with ID starting with b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325 not found: ID does not exist" containerID="b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325" Jul 10 19:25:45.263043 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.262991 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325"} err="failed to get container status \"b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325\": rpc error: code = NotFound desc = could not find container \"b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325\": container with ID starting with b60e8b80855f7bcc5c099bd65b8ca17bbf787d3e51586ebb4378838a1d8a3325 not found: ID does not exist" Jul 10 19:25:45.314427 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.314398 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95a108a-eead-4733-81a3-132be8780409-maas-api-tls\") pod \"f95a108a-eead-4733-81a3-132be8780409\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " Jul 10 19:25:45.314569 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.314496 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95a108a-eead-4733-81a3-132be8780409-openshift-service-ca\") pod \"f95a108a-eead-4733-81a3-132be8780409\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " Jul 10 19:25:45.314569 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.314538 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxs74\" (UniqueName: \"kubernetes.io/projected/f95a108a-eead-4733-81a3-132be8780409-kube-api-access-qxs74\") pod \"f95a108a-eead-4733-81a3-132be8780409\" (UID: \"f95a108a-eead-4733-81a3-132be8780409\") " Jul 10 19:25:45.314901 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.314878 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95a108a-eead-4733-81a3-132be8780409-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f95a108a-eead-4733-81a3-132be8780409" (UID: "f95a108a-eead-4733-81a3-132be8780409"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:25:45.316720 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.316695 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95a108a-eead-4733-81a3-132be8780409-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f95a108a-eead-4733-81a3-132be8780409" (UID: "f95a108a-eead-4733-81a3-132be8780409"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:25:45.316841 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.316752 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95a108a-eead-4733-81a3-132be8780409-kube-api-access-qxs74" (OuterVolumeSpecName: "kube-api-access-qxs74") pod "f95a108a-eead-4733-81a3-132be8780409" (UID: "f95a108a-eead-4733-81a3-132be8780409"). InnerVolumeSpecName "kube-api-access-qxs74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:45.416218 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.416181 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f95a108a-eead-4733-81a3-132be8780409-openshift-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:45.416218 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.416210 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxs74\" (UniqueName: \"kubernetes.io/projected/f95a108a-eead-4733-81a3-132be8780409-kube-api-access-qxs74\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:45.416218 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.416220 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f95a108a-eead-4733-81a3-132be8780409-maas-api-tls\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:45.560269 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.560238 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4"] Jul 10 19:25:45.563141 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.563109 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c05b0410-7f99669999-9dvm4"] Jul 10 19:25:45.805626 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:45.805528 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f95a108a-eead-4733-81a3-132be8780409" path="/var/lib/kubelet/pods/f95a108a-eead-4733-81a3-132be8780409/volumes" Jul 10 19:25:50.516744 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:50.516713 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb"] Jul 10 19:25:50.517148 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:50.516919 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" podUID="91b454ba-b5df-42d0-a41e-eba92ce609b6" containerName="istio-proxy" containerID="cri-o://de9a464c054fb7c8cc7185d67a2d3b7dcdb01c8c2fc87d5d82fbe2ce2a66795d" gracePeriod=30 Jul 10 19:25:51.269833 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.269783 2549 generic.go:356] "Generic (PLEG): container finished" podID="91b454ba-b5df-42d0-a41e-eba92ce609b6" containerID="de9a464c054fb7c8cc7185d67a2d3b7dcdb01c8c2fc87d5d82fbe2ce2a66795d" exitCode=0 Jul 10 19:25:51.270012 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.269832 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" event={"ID":"91b454ba-b5df-42d0-a41e-eba92ce609b6","Type":"ContainerDied","Data":"de9a464c054fb7c8cc7185d67a2d3b7dcdb01c8c2fc87d5d82fbe2ce2a66795d"} Jul 10 19:25:51.376243 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.376216 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:51.481279 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481252 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b454ba-b5df-42d0-a41e-eba92ce609b6-istiod-ca-cert\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481423 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481300 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-socket\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481423 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481329 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-podinfo\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481423 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481367 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-token\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481423 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481410 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-certs\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481636 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481442 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-credential-socket\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481636 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481466 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-envoy\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481636 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481519 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mthrf\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-kube-api-access-mthrf\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481636 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481556 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-data\") pod \"91b454ba-b5df-42d0-a41e-eba92ce609b6\" (UID: \"91b454ba-b5df-42d0-a41e-eba92ce609b6\") " Jul 10 19:25:51.481876 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481706 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:51.481876 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481714 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:51.481876 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481705 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b454ba-b5df-42d0-a41e-eba92ce609b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:25:51.481876 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481735 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:51.482073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481963 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:51.482073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.481985 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:51.482073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.482002 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91b454ba-b5df-42d0-a41e-eba92ce609b6-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:51.482073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.482014 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:51.482722 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.482698 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:51.483672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.483651 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:25:51.483749 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.483726 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:51.484177 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.484156 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:25:51.484177 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.484159 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-kube-api-access-mthrf" (OuterVolumeSpecName: "kube-api-access-mthrf") pod "91b454ba-b5df-42d0-a41e-eba92ce609b6" (UID: "91b454ba-b5df-42d0-a41e-eba92ce609b6"). InnerVolumeSpecName "kube-api-access-mthrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:25:51.583074 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.583047 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:51.583074 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.583069 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:51.583074 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.583078 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mthrf\" (UniqueName: \"kubernetes.io/projected/91b454ba-b5df-42d0-a41e-eba92ce609b6-kube-api-access-mthrf\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:51.583485 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.583086 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:51.583485 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:51.583095 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91b454ba-b5df-42d0-a41e-eba92ce609b6-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:25:52.276525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:52.276443 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" Jul 10 19:25:52.276525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:52.276463 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb" event={"ID":"91b454ba-b5df-42d0-a41e-eba92ce609b6","Type":"ContainerDied","Data":"6a1c49c3908504dcf60440a0fbf1b08a0fded95488b51200cb8367aa4aed2cf1"} Jul 10 19:25:52.276525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:52.276524 2549 scope.go:117] "RemoveContainer" containerID="de9a464c054fb7c8cc7185d67a2d3b7dcdb01c8c2fc87d5d82fbe2ce2a66795d" Jul 10 19:25:52.294581 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:52.294557 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb"] Jul 10 19:25:52.300940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:52.300916 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c05b0410-openshift-default-86dcfbcf49-ts8bb"] Jul 10 19:25:53.806198 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:53.806167 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91b454ba-b5df-42d0-a41e-eba92ce609b6" path="/var/lib/kubelet/pods/91b454ba-b5df-42d0-a41e-eba92ce609b6/volumes" Jul 10 19:25:57.104212 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.104181 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45"] Jul 10 19:25:57.104710 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.104690 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f95a108a-eead-4733-81a3-132be8780409" containerName="maas-api" Jul 10 19:25:57.104814 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.104713 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f95a108a-eead-4733-81a3-132be8780409" containerName="maas-api" Jul 10 19:25:57.104814 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.104757 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b454ba-b5df-42d0-a41e-eba92ce609b6" containerName="istio-proxy" Jul 10 19:25:57.104814 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.104766 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b454ba-b5df-42d0-a41e-eba92ce609b6" containerName="istio-proxy" Jul 10 19:25:57.104965 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.104881 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="91b454ba-b5df-42d0-a41e-eba92ce609b6" containerName="istio-proxy" Jul 10 19:25:57.104965 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.104896 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="f95a108a-eead-4733-81a3-132be8780409" containerName="maas-api" Jul 10 19:25:57.108209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.108187 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.110732 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.110713 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-eec4917b-openshift-default-dockercfg-fkcwx\"" Jul 10 19:25:57.119565 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.119544 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45"] Jul 10 19:25:57.234383 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234351 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-data\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.234561 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234390 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrgmn\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-kube-api-access-mrgmn\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.234561 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234411 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330d7a82-4cfb-41d1-8ed4-082a88914652-istiod-ca-cert\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.234561 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234483 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-credential-socket\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.234561 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234502 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-socket\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.234561 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234518 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-podinfo\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.234760 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234568 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-envoy\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.234760 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234609 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-token\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.234760 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.234629 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-certs\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.335439 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335405 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-certs\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.335637 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335447 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-data\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.335637 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335587 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrgmn\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-kube-api-access-mrgmn\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.335637 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335617 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330d7a82-4cfb-41d1-8ed4-082a88914652-istiod-ca-cert\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.335839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335668 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-credential-socket\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.335839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335700 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-socket\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.335839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335722 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-podinfo\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.336008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335867 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-data\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.336008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335918 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-envoy\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.336008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335982 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-token\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.336157 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.336110 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-socket\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.336157 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.335917 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-certs\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.336265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.336182 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-credential-socket\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.336460 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.336437 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330d7a82-4cfb-41d1-8ed4-082a88914652-istiod-ca-cert\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.338723 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.338700 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-envoy\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.338951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.338936 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-podinfo\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.346268 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.346244 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-token\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.346438 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.346419 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrgmn\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-kube-api-access-mrgmn\") pod \"e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.422362 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.422291 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:57.567072 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.567046 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45"] Jul 10 19:25:57.569305 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:25:57.569276 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330d7a82_4cfb_41d1_8ed4_082a88914652.slice/crio-82416b2287c4e2d68a1f2948922c17bf1e6012575fd2dd3493925ac07c9ec1e1 WatchSource:0}: Error finding container 82416b2287c4e2d68a1f2948922c17bf1e6012575fd2dd3493925ac07c9ec1e1: Status 404 returned error can't find the container with id 82416b2287c4e2d68a1f2948922c17bf1e6012575fd2dd3493925ac07c9ec1e1 Jul 10 19:25:57.571454 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.571420 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:57.571537 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.571486 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:57.571593 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:57.571555 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:25:58.307199 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.307151 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" event={"ID":"330d7a82-4cfb-41d1-8ed4-082a88914652","Type":"ContainerStarted","Data":"4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7"} Jul 10 19:25:58.307581 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.307214 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" event={"ID":"330d7a82-4cfb-41d1-8ed4-082a88914652","Type":"ContainerStarted","Data":"82416b2287c4e2d68a1f2948922c17bf1e6012575fd2dd3493925ac07c9ec1e1"} Jul 10 19:25:58.332551 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.332492 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" podStartSLOduration=1.332472572 podStartE2EDuration="1.332472572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:58.328672557 +0000 UTC m=+1535.079577440" watchObservedRunningTime="2026-07-10 19:25:58.332472572 +0000 UTC m=+1535.083377448" Jul 10 19:25:58.393485 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.393452 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9"] Jul 10 19:25:58.397209 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.397189 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.399458 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.399436 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-eec4917b-serving-cert\"" Jul 10 19:25:58.408749 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.408728 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9"] Jul 10 19:25:58.423195 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.423170 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:58.428092 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.428064 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:58.548008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.547977 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-maas-api-tls\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.548179 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.548014 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-openshift-service-ca\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.548179 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.548089 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s78nc\" (UniqueName: \"kubernetes.io/projected/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-kube-api-access-s78nc\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.649045 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.649007 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-maas-api-tls\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.649223 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.649049 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-openshift-service-ca\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.649223 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.649130 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s78nc\" (UniqueName: \"kubernetes.io/projected/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-kube-api-access-s78nc\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.649755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.649735 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-openshift-service-ca\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.651448 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.651428 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-maas-api-tls\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.667675 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.667652 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s78nc\" (UniqueName: \"kubernetes.io/projected/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-kube-api-access-s78nc\") pod \"maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.708317 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.708277 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:58.846660 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:58.846635 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9"] Jul 10 19:25:58.848632 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:25:58.848610 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fcd3b6_1f42_4b35_b2c4_3905e7e5bcc8.slice/crio-a9185fc9680b3f5349a06d43134167c25ef863313e1bf578f7d5822f6c4f739e WatchSource:0}: Error finding container a9185fc9680b3f5349a06d43134167c25ef863313e1bf578f7d5822f6c4f739e: Status 404 returned error can't find the container with id a9185fc9680b3f5349a06d43134167c25ef863313e1bf578f7d5822f6c4f739e Jul 10 19:25:59.313720 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:59.313687 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" event={"ID":"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8","Type":"ContainerStarted","Data":"c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5"} Jul 10 19:25:59.314162 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:59.313909 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:25:59.314162 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:59.313950 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:59.314162 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:59.313987 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" event={"ID":"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8","Type":"ContainerStarted","Data":"a9185fc9680b3f5349a06d43134167c25ef863313e1bf578f7d5822f6c4f739e"} Jul 10 19:25:59.314937 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:59.314913 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:25:59.335130 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:25:59.335074 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" podStartSLOduration=1.011967511 podStartE2EDuration="1.335054143s" totalImagesPullingTime="323.086632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:59.332405331 +0000 UTC m=+1536.083310207" watchObservedRunningTime="2026-07-10 19:25:59.335054143 +0000 UTC m=+1536.085959019" Jul 10 19:26:02.824736 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:02.824699 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45"] Jul 10 19:26:02.825275 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:02.824913 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" podUID="330d7a82-4cfb-41d1-8ed4-082a88914652" containerName="istio-proxy" containerID="cri-o://4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7" gracePeriod=30 Jul 10 19:26:03.030333 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.030297 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9"] Jul 10 19:26:03.030898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.030867 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" podUID="50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" containerName="maas-api" containerID="cri-o://c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5" gracePeriod=30 Jul 10 19:26:03.285278 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.285253 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:26:03.335583 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.335551 2549 generic.go:356] "Generic (PLEG): container finished" podID="50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" containerID="c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5" exitCode=0 Jul 10 19:26:03.335748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.335605 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" Jul 10 19:26:03.335748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.335608 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" event={"ID":"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8","Type":"ContainerDied","Data":"c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5"} Jul 10 19:26:03.335851 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.335756 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9" event={"ID":"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8","Type":"ContainerDied","Data":"a9185fc9680b3f5349a06d43134167c25ef863313e1bf578f7d5822f6c4f739e"} Jul 10 19:26:03.335851 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.335813 2549 scope.go:117] "RemoveContainer" containerID="c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5" Jul 10 19:26:03.345012 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.344984 2549 scope.go:117] "RemoveContainer" containerID="c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5" Jul 10 19:26:03.345262 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:26:03.345242 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5\": container with ID starting with c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5 not found: ID does not exist" containerID="c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5" Jul 10 19:26:03.345314 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.345272 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5"} err="failed to get container status \"c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5\": rpc error: code = NotFound desc = could not find container \"c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5\": container with ID starting with c672943f84eb820b3e43b475b469dfc89c18eb925432f558d4cf7b6f4ba3fef5 not found: ID does not exist" Jul 10 19:26:03.393071 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.393044 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s78nc\" (UniqueName: \"kubernetes.io/projected/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-kube-api-access-s78nc\") pod \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " Jul 10 19:26:03.393251 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.393144 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-maas-api-tls\") pod \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " Jul 10 19:26:03.393251 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.393194 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-openshift-service-ca\") pod \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\" (UID: \"50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8\") " Jul 10 19:26:03.393547 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.393519 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" (UID: "50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:26:03.395131 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.395096 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-kube-api-access-s78nc" (OuterVolumeSpecName: "kube-api-access-s78nc") pod "50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" (UID: "50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8"). InnerVolumeSpecName "kube-api-access-s78nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:26:03.395227 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.395167 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" (UID: "50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:26:03.494413 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.494385 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-maas-api-tls\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:03.494413 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.494411 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-openshift-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:03.494413 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.494422 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s78nc\" (UniqueName: \"kubernetes.io/projected/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8-kube-api-access-s78nc\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:03.658952 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.658873 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9"] Jul 10 19:26:03.660632 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.660613 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eec4917b-559fc7f84f-ncbd9"] Jul 10 19:26:03.806535 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:03.806505 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" path="/var/lib/kubelet/pods/50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8/volumes" Jul 10 19:26:08.093947 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.093883 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:26:08.236082 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236043 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-certs\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236285 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236105 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-data\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236285 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236126 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrgmn\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-kube-api-access-mrgmn\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236285 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236146 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-podinfo\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236285 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236273 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-token\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236327 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-credential-socket\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236350 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-socket\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236384 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330d7a82-4cfb-41d1-8ed4-082a88914652-istiod-ca-cert\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236446 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-envoy\") pod \"330d7a82-4cfb-41d1-8ed4-082a88914652\" (UID: \"330d7a82-4cfb-41d1-8ed4-082a88914652\") " Jul 10 19:26:08.236514 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236460 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:08.236773 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236610 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:08.236773 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236624 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:08.236928 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236808 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330d7a82-4cfb-41d1-8ed4-082a88914652-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:26:08.236928 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236869 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.236928 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236895 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.237058 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236971 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-data" (OuterVolumeSpecName: "istio-data") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:08.237058 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.236984 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.238533 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.238498 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-kube-api-access-mrgmn" (OuterVolumeSpecName: "kube-api-access-mrgmn") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "kube-api-access-mrgmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:26:08.238533 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.238511 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:26:08.238533 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.238518 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-token" (OuterVolumeSpecName: "istio-token") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:26:08.238742 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.238727 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "330d7a82-4cfb-41d1-8ed4-082a88914652" (UID: "330d7a82-4cfb-41d1-8ed4-082a88914652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:26:08.338558 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.338520 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.338558 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.338558 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrgmn\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-kube-api-access-mrgmn\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.338763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.338574 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.338763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.338588 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.338763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.338601 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/330d7a82-4cfb-41d1-8ed4-082a88914652-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.338763 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.338613 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/330d7a82-4cfb-41d1-8ed4-082a88914652-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:26:08.361751 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.361663 2549 generic.go:356] "Generic (PLEG): container finished" podID="330d7a82-4cfb-41d1-8ed4-082a88914652" containerID="4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7" exitCode=0 Jul 10 19:26:08.361751 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.361733 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" Jul 10 19:26:08.361957 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.361744 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" event={"ID":"330d7a82-4cfb-41d1-8ed4-082a88914652","Type":"ContainerDied","Data":"4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7"} Jul 10 19:26:08.361957 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.361839 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45" event={"ID":"330d7a82-4cfb-41d1-8ed4-082a88914652","Type":"ContainerDied","Data":"82416b2287c4e2d68a1f2948922c17bf1e6012575fd2dd3493925ac07c9ec1e1"} Jul 10 19:26:08.361957 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.361863 2549 scope.go:117] "RemoveContainer" containerID="4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7" Jul 10 19:26:08.371271 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.371255 2549 scope.go:117] "RemoveContainer" containerID="4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7" Jul 10 19:26:08.371525 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:26:08.371497 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7\": container with ID starting with 4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7 not found: ID does not exist" containerID="4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7" Jul 10 19:26:08.371577 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.371533 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7"} err="failed to get container status \"4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7\": rpc error: code = NotFound desc = could not find container \"4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7\": container with ID starting with 4f28c7addc2f4be74d710a042ba29f763fe298746c4a57cb3f9713b612c77bf7 not found: ID does not exist" Jul 10 19:26:08.382680 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.382659 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45"] Jul 10 19:26:08.385902 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:08.385882 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-eec4917b-openshift-default-6c494cfb95-vlk45"] Jul 10 19:26:09.805570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:26:09.805534 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="330d7a82-4cfb-41d1-8ed4-082a88914652" path="/var/lib/kubelet/pods/330d7a82-4cfb-41d1-8ed4-082a88914652/volumes" Jul 10 19:27:45.441814 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.441768 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6"] Jul 10 19:27:45.442265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.442158 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330d7a82-4cfb-41d1-8ed4-082a88914652" containerName="istio-proxy" Jul 10 19:27:45.442265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.442170 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="330d7a82-4cfb-41d1-8ed4-082a88914652" containerName="istio-proxy" Jul 10 19:27:45.442265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.442183 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" containerName="maas-api" Jul 10 19:27:45.442265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.442189 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" containerName="maas-api" Jul 10 19:27:45.442265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.442251 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="330d7a82-4cfb-41d1-8ed4-082a88914652" containerName="istio-proxy" Jul 10 19:27:45.442265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.442263 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fcd3b6-1f42-4b35-b2c4-3905e7e5bcc8" containerName="maas-api" Jul 10 19:27:45.445305 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.445283 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.448014 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.447989 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-379e27ec-openshift-default-dockercfg-klskz\"" Jul 10 19:27:45.466434 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.466399 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6"] Jul 10 19:27:45.600596 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600547 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-socket\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.600596 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600598 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istiod-ca-cert\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.600866 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600715 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-credential-socket\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.600866 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600830 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-data\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.600866 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600856 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-token\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.600999 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600878 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-envoy\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.600999 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600899 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftfk\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-kube-api-access-pftfk\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.600999 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600930 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-certs\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.600999 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.600963 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-podinfo\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.702519 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702430 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-socket\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.702519 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702472 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istiod-ca-cert\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.702519 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702516 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-credential-socket\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.702746 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702568 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-data\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.702746 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702703 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-token\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.702889 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702756 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-envoy\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.702889 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702824 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftfk\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-kube-api-access-pftfk\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.702889 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702877 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-certs\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.703026 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702898 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-podinfo\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.703026 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702928 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-data\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.703026 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.702965 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-socket\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.703185 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.703025 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-credential-socket\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.703245 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.703225 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-certs\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.703353 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.703333 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istiod-ca-cert\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.705315 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.705291 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-envoy\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.705503 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.705489 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-podinfo\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.710781 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.710761 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-token\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.711075 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.711055 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftfk\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-kube-api-access-pftfk\") pod \"e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.756132 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.755840 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:45.898932 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.898901 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6"] Jul 10 19:27:45.901616 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:27:45.901591 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ff7873_5f4d_4dab_b2eb_53f35ee6e4fc.slice/crio-8162d29b111fc13c57527c61b2e762fc4e9184c7cc6b064d9c9ad2f955e6d7e5 WatchSource:0}: Error finding container 8162d29b111fc13c57527c61b2e762fc4e9184c7cc6b064d9c9ad2f955e6d7e5: Status 404 returned error can't find the container with id 8162d29b111fc13c57527c61b2e762fc4e9184c7cc6b064d9c9ad2f955e6d7e5 Jul 10 19:27:45.904171 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.904132 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:27:45.904371 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.904346 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:27:45.904522 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:45.904496 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:27:46.859154 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:46.859115 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" event={"ID":"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc","Type":"ContainerStarted","Data":"217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63"} Jul 10 19:27:46.859608 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:46.859170 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" event={"ID":"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc","Type":"ContainerStarted","Data":"8162d29b111fc13c57527c61b2e762fc4e9184c7cc6b064d9c9ad2f955e6d7e5"} Jul 10 19:27:46.880876 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:46.880822 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" podStartSLOduration=1.880785591 podStartE2EDuration="1.880785591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:27:46.879015633 +0000 UTC m=+1643.629920536" watchObservedRunningTime="2026-07-10 19:27:46.880785591 +0000 UTC m=+1643.631690535" Jul 10 19:27:47.757278 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:47.757237 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:47.762266 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:47.762243 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:47.864889 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:47.864859 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:47.865738 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:47.865721 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:27:48.833810 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:48.833762 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp"] Jul 10 19:27:48.837969 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:48.837945 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:48.840464 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:48.840443 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-379e27ec-serving-cert\"" Jul 10 19:27:48.846666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:48.846643 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp"] Jul 10 19:27:48.933937 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:48.933904 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n48db\" (UniqueName: \"kubernetes.io/projected/ead9b299-166e-4cdf-9c3c-a023c81cb428-kube-api-access-n48db\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:48.934333 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:48.933949 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead9b299-166e-4cdf-9c3c-a023c81cb428-openshift-service-ca\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:48.934333 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:48.933995 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead9b299-166e-4cdf-9c3c-a023c81cb428-maas-api-tls\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:49.035101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.035067 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n48db\" (UniqueName: \"kubernetes.io/projected/ead9b299-166e-4cdf-9c3c-a023c81cb428-kube-api-access-n48db\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:49.035101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.035110 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead9b299-166e-4cdf-9c3c-a023c81cb428-openshift-service-ca\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:49.035341 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.035264 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead9b299-166e-4cdf-9c3c-a023c81cb428-maas-api-tls\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:49.036022 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.035998 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead9b299-166e-4cdf-9c3c-a023c81cb428-openshift-service-ca\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:49.038138 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.038118 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead9b299-166e-4cdf-9c3c-a023c81cb428-maas-api-tls\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:49.044190 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.044172 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n48db\" (UniqueName: \"kubernetes.io/projected/ead9b299-166e-4cdf-9c3c-a023c81cb428-kube-api-access-n48db\") pod \"maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:49.150586 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.150494 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:49.309595 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.309568 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp"] Jul 10 19:27:49.311403 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:27:49.311374 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead9b299_166e_4cdf_9c3c_a023c81cb428.slice/crio-2d817925ee5c0e888262e4ddb1c7da17654a70f7a9542a97170fb6b497335c3c WatchSource:0}: Error finding container 2d817925ee5c0e888262e4ddb1c7da17654a70f7a9542a97170fb6b497335c3c: Status 404 returned error can't find the container with id 2d817925ee5c0e888262e4ddb1c7da17654a70f7a9542a97170fb6b497335c3c Jul 10 19:27:49.876614 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:49.876581 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" event={"ID":"ead9b299-166e-4cdf-9c3c-a023c81cb428","Type":"ContainerStarted","Data":"2d817925ee5c0e888262e4ddb1c7da17654a70f7a9542a97170fb6b497335c3c"} Jul 10 19:27:50.883629 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:50.883597 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" event={"ID":"ead9b299-166e-4cdf-9c3c-a023c81cb428","Type":"ContainerStarted","Data":"9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296"} Jul 10 19:27:50.884060 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:50.883708 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:27:50.902120 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:50.902072 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" podStartSLOduration=2.365244629 podStartE2EDuration="2.902054194s" totalImagesPullingTime="536.809565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:27:50.899575584 +0000 UTC m=+1647.650480487" watchObservedRunningTime="2026-07-10 19:27:50.902054194 +0000 UTC m=+1647.652959069" Jul 10 19:27:56.894608 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:27:56.894580 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:28:00.943666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:00.943618 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp"] Jul 10 19:28:00.944508 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:00.944474 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" podUID="ead9b299-166e-4cdf-9c3c-a023c81cb428" containerName="maas-api" containerID="cri-o://9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296" gracePeriod=30 Jul 10 19:28:01.189375 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.189352 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:28:01.350901 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.350866 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead9b299-166e-4cdf-9c3c-a023c81cb428-openshift-service-ca\") pod \"ead9b299-166e-4cdf-9c3c-a023c81cb428\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " Jul 10 19:28:01.351072 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.350922 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n48db\" (UniqueName: \"kubernetes.io/projected/ead9b299-166e-4cdf-9c3c-a023c81cb428-kube-api-access-n48db\") pod \"ead9b299-166e-4cdf-9c3c-a023c81cb428\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " Jul 10 19:28:01.351072 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.351027 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead9b299-166e-4cdf-9c3c-a023c81cb428-maas-api-tls\") pod \"ead9b299-166e-4cdf-9c3c-a023c81cb428\" (UID: \"ead9b299-166e-4cdf-9c3c-a023c81cb428\") " Jul 10 19:28:01.351314 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.351286 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead9b299-166e-4cdf-9c3c-a023c81cb428-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ead9b299-166e-4cdf-9c3c-a023c81cb428" (UID: "ead9b299-166e-4cdf-9c3c-a023c81cb428"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:28:01.353274 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.353229 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead9b299-166e-4cdf-9c3c-a023c81cb428-kube-api-access-n48db" (OuterVolumeSpecName: "kube-api-access-n48db") pod "ead9b299-166e-4cdf-9c3c-a023c81cb428" (UID: "ead9b299-166e-4cdf-9c3c-a023c81cb428"). InnerVolumeSpecName "kube-api-access-n48db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:28:01.353415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.353379 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead9b299-166e-4cdf-9c3c-a023c81cb428-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ead9b299-166e-4cdf-9c3c-a023c81cb428" (UID: "ead9b299-166e-4cdf-9c3c-a023c81cb428"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:28:01.452198 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.452153 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead9b299-166e-4cdf-9c3c-a023c81cb428-maas-api-tls\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:01.452198 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.452185 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead9b299-166e-4cdf-9c3c-a023c81cb428-openshift-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:01.452198 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.452197 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n48db\" (UniqueName: \"kubernetes.io/projected/ead9b299-166e-4cdf-9c3c-a023c81cb428-kube-api-access-n48db\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:01.945060 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.943582 2549 generic.go:356] "Generic (PLEG): container finished" podID="ead9b299-166e-4cdf-9c3c-a023c81cb428" containerID="9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296" exitCode=0 Jul 10 19:28:01.945060 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.943634 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" Jul 10 19:28:01.945060 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.943666 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" event={"ID":"ead9b299-166e-4cdf-9c3c-a023c81cb428","Type":"ContainerDied","Data":"9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296"} Jul 10 19:28:01.945060 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.943728 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp" event={"ID":"ead9b299-166e-4cdf-9c3c-a023c81cb428","Type":"ContainerDied","Data":"2d817925ee5c0e888262e4ddb1c7da17654a70f7a9542a97170fb6b497335c3c"} Jul 10 19:28:01.945060 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.943753 2549 scope.go:117] "RemoveContainer" containerID="9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296" Jul 10 19:28:01.954267 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.954087 2549 scope.go:117] "RemoveContainer" containerID="9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296" Jul 10 19:28:01.954373 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:28:01.954354 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296\": container with ID starting with 9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296 not found: ID does not exist" containerID="9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296" Jul 10 19:28:01.954414 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.954384 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296"} err="failed to get container status \"9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296\": rpc error: code = NotFound desc = could not find container \"9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296\": container with ID starting with 9f6778fba968c1615eab5a721e16b5405448639c5c2263cd1417104fe2a71296 not found: ID does not exist" Jul 10 19:28:01.961998 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.961977 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp"] Jul 10 19:28:01.967607 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:01.967586 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-379e27ec-5ddf986f6d-bnnzp"] Jul 10 19:28:02.866441 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:02.866403 2549 patch_prober.go:28] interesting pod/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 10 19:28:02.866639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:02.866476 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 10 19:28:02.950894 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:02.950861 2549 generic.go:356] "Generic (PLEG): container finished" podID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerID="217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63" exitCode=0 Jul 10 19:28:02.951300 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:02.950933 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" event={"ID":"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc","Type":"ContainerDied","Data":"217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63"} Jul 10 19:28:02.951387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:02.951371 2549 scope.go:117] "RemoveContainer" containerID="217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63" Jul 10 19:28:02.952406 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:02.952371 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:02.952509 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:02.952446 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:02.952509 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:02.952485 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:03.808013 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:03.807980 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ead9b299-166e-4cdf-9c3c-a023c81cb428" path="/var/lib/kubelet/pods/ead9b299-166e-4cdf-9c3c-a023c81cb428/volumes" Jul 10 19:28:03.958685 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:03.958650 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" event={"ID":"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc","Type":"ContainerStarted","Data":"5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610"} Jul 10 19:28:04.757356 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:04.757320 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:28:04.757755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:04.757631 2549 patch_prober.go:28] interesting pod/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 10 19:28:04.757952 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:04.757929 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 10 19:28:05.757326 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:05.757282 2549 patch_prober.go:28] interesting pod/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 10 19:28:05.757856 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:05.757342 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 10 19:28:06.756534 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:06.756497 2549 patch_prober.go:28] interesting pod/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 10 19:28:06.756710 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:06.756563 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 10 19:28:07.066742 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:07.066706 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6"] Jul 10 19:28:07.067249 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:07.066944 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" containerID="cri-o://5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610" gracePeriod=30 Jul 10 19:28:13.431487 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.431463 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:28:13.452587 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.452562 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-envoy\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.452702 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.452598 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-data\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.452702 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.452617 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pftfk\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-kube-api-access-pftfk\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.452702 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.452640 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istiod-ca-cert\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.452702 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.452679 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-podinfo\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.452931 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.452738 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-socket\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.452931 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.452875 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-token\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.453038 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.452984 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-certs\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.453097 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.453034 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-credential-socket\") pod \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\" (UID: \"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc\") " Jul 10 19:28:13.453696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.453167 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:28:13.453696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.453372 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:13.453696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.453523 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:13.453696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.453568 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:13.453696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.453643 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:13.454074 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.453834 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:13.454074 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.453853 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:13.455421 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.455384 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:28:13.455694 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.455662 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:28:13.455833 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.455786 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-kube-api-access-pftfk" (OuterVolumeSpecName: "kube-api-access-pftfk") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "kube-api-access-pftfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:28:13.455907 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.455874 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" (UID: "67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:28:13.554387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.554293 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:13.554387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.554324 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:13.554387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.554338 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:13.554387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.554349 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:13.554387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.554360 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:13.554387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.554372 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pftfk\" (UniqueName: \"kubernetes.io/projected/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-kube-api-access-pftfk\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:13.554387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:13.554383 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:28:14.008776 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.008739 2549 generic.go:356] "Generic (PLEG): container finished" podID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerID="5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610" exitCode=0 Jul 10 19:28:14.008967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.008787 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" Jul 10 19:28:14.008967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.008826 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" event={"ID":"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc","Type":"ContainerDied","Data":"5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610"} Jul 10 19:28:14.008967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.008885 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6" event={"ID":"67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc","Type":"ContainerDied","Data":"8162d29b111fc13c57527c61b2e762fc4e9184c7cc6b064d9c9ad2f955e6d7e5"} Jul 10 19:28:14.008967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.008912 2549 scope.go:117] "RemoveContainer" containerID="5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610" Jul 10 19:28:14.017751 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.017730 2549 scope.go:117] "RemoveContainer" containerID="217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63" Jul 10 19:28:14.026018 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.025999 2549 scope.go:117] "RemoveContainer" containerID="5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610" Jul 10 19:28:14.026313 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:28:14.026279 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610\": container with ID starting with 5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610 not found: ID does not exist" containerID="5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610" Jul 10 19:28:14.026406 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.026314 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610"} err="failed to get container status \"5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610\": rpc error: code = NotFound desc = could not find container \"5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610\": container with ID starting with 5ee7cb76349c76296b40e6671e054c2e3203466b2bf790a1263c5d8c41727610 not found: ID does not exist" Jul 10 19:28:14.026406 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.026338 2549 scope.go:117] "RemoveContainer" containerID="217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63" Jul 10 19:28:14.026641 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:28:14.026613 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63\": container with ID starting with 217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63 not found: ID does not exist" containerID="217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63" Jul 10 19:28:14.026699 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.026649 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63"} err="failed to get container status \"217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63\": rpc error: code = NotFound desc = could not find container \"217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63\": container with ID starting with 217e4639ec2f8533d8ecc56966d26c54af839e3229a253f3fc085e0755b07f63 not found: ID does not exist" Jul 10 19:28:14.027911 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.027890 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6"] Jul 10 19:28:14.031880 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:14.031857 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-379e27ec-openshift-default-6dc6cb9f9-htwr6"] Jul 10 19:28:15.805056 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:15.805021 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" path="/var/lib/kubelet/pods/67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc/volumes" Jul 10 19:28:26.797369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797333 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv"] Jul 10 19:28:26.797750 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797738 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead9b299-166e-4cdf-9c3c-a023c81cb428" containerName="maas-api" Jul 10 19:28:26.797822 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797752 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead9b299-166e-4cdf-9c3c-a023c81cb428" containerName="maas-api" Jul 10 19:28:26.797822 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797769 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" Jul 10 19:28:26.797822 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797775 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" Jul 10 19:28:26.797822 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797786 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" Jul 10 19:28:26.797822 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797809 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" Jul 10 19:28:26.797978 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797868 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" Jul 10 19:28:26.797978 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797875 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ff7873-5f4d-4dab-b2eb-53f35ee6e4fc" containerName="istio-proxy" Jul 10 19:28:26.797978 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.797882 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead9b299-166e-4cdf-9c3c-a023c81cb428" containerName="maas-api" Jul 10 19:28:26.802159 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.802125 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.804700 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.804678 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3d2538db-openshift-default-dockercfg-rvjpr\"" Jul 10 19:28:26.814948 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.814921 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv"] Jul 10 19:28:26.875284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875255 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-data\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.875430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875290 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks27t\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-kube-api-access-ks27t\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.875430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875324 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-credential-socket\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.875430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875349 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-podinfo\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.875570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875450 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-socket\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.875570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875481 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istiod-ca-cert\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.875570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875515 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-token\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.875570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875537 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-certs\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.875570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.875552 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-envoy\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976147 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976103 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-token\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976147 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976153 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-certs\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976169 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-envoy\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976230 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-data\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976263 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks27t\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-kube-api-access-ks27t\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976415 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976293 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-credential-socket\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976634 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976424 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-podinfo\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976634 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976560 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-socket\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976634 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976579 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-certs\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976634 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976590 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istiod-ca-cert\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976827 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976688 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-data\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976827 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976698 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-credential-socket\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.976905 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.976857 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-socket\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.977220 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.977201 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istiod-ca-cert\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.978697 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.978680 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-envoy\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.979042 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.979021 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-podinfo\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.983982 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.983960 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-token\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:26.984201 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:26.984181 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks27t\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-kube-api-access-ks27t\") pod \"e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:27.113779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:27.113727 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:27.263627 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:27.263603 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv"] Jul 10 19:28:27.265872 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:28:27.265845 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b627fe5_3df1_4493_acb3_4e7f9bae83b2.slice/crio-37c00faf24eac2c5ecb42b16dc1509ce1471a09d2ac986875c9083fd2677b1c2 WatchSource:0}: Error finding container 37c00faf24eac2c5ecb42b16dc1509ce1471a09d2ac986875c9083fd2677b1c2: Status 404 returned error can't find the container with id 37c00faf24eac2c5ecb42b16dc1509ce1471a09d2ac986875c9083fd2677b1c2 Jul 10 19:28:27.268063 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:27.268031 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:27.268166 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:27.268094 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:27.268166 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:27.268121 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:28.086285 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:28.086239 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" event={"ID":"0b627fe5-3df1-4493-acb3-4e7f9bae83b2","Type":"ContainerStarted","Data":"16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3"} Jul 10 19:28:28.086677 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:28.086337 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" event={"ID":"0b627fe5-3df1-4493-acb3-4e7f9bae83b2","Type":"ContainerStarted","Data":"37c00faf24eac2c5ecb42b16dc1509ce1471a09d2ac986875c9083fd2677b1c2"} Jul 10 19:28:28.106087 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:28.105969 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" podStartSLOduration=2.105951889 podStartE2EDuration="2.105951889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:28:28.105004912 +0000 UTC m=+1684.855909827" watchObservedRunningTime="2026-07-10 19:28:28.105951889 +0000 UTC m=+1684.856856763" Jul 10 19:28:28.114714 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:28.114676 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:28.119825 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:28.119804 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:29.091924 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:29.091896 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:29.092919 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:29.092900 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:30.186374 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.186339 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf"] Jul 10 19:28:30.190552 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.190531 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.193090 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.193066 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d2538db-serving-cert\"" Jul 10 19:28:30.202231 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.202209 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf"] Jul 10 19:28:30.305782 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.305750 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b52\" (UniqueName: \"kubernetes.io/projected/33498dd8-d969-4075-a1c7-479ee0ceaed4-kube-api-access-99b52\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.305967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.305785 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33498dd8-d969-4075-a1c7-479ee0ceaed4-openshift-service-ca\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.305967 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.305890 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33498dd8-d969-4075-a1c7-479ee0ceaed4-maas-api-tls\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.406947 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.406915 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33498dd8-d969-4075-a1c7-479ee0ceaed4-maas-api-tls\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.407133 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.407041 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b52\" (UniqueName: \"kubernetes.io/projected/33498dd8-d969-4075-a1c7-479ee0ceaed4-kube-api-access-99b52\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.407133 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.407068 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33498dd8-d969-4075-a1c7-479ee0ceaed4-openshift-service-ca\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.407710 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.407682 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33498dd8-d969-4075-a1c7-479ee0ceaed4-openshift-service-ca\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.409445 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.409427 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33498dd8-d969-4075-a1c7-479ee0ceaed4-maas-api-tls\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.417354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.417334 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b52\" (UniqueName: \"kubernetes.io/projected/33498dd8-d969-4075-a1c7-479ee0ceaed4-kube-api-access-99b52\") pod \"maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.502907 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.502821 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:30.659679 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:30.659651 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf"] Jul 10 19:28:30.662589 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:28:30.662548 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33498dd8_d969_4075_a1c7_479ee0ceaed4.slice/crio-0a06166855d124af9852c83c0e6d318ef0e9992a5acc7dcc31070bac843fb8e1 WatchSource:0}: Error finding container 0a06166855d124af9852c83c0e6d318ef0e9992a5acc7dcc31070bac843fb8e1: Status 404 returned error can't find the container with id 0a06166855d124af9852c83c0e6d318ef0e9992a5acc7dcc31070bac843fb8e1 Jul 10 19:28:31.102455 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:31.102423 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" event={"ID":"33498dd8-d969-4075-a1c7-479ee0ceaed4","Type":"ContainerStarted","Data":"ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7"} Jul 10 19:28:31.102582 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:31.102484 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" event={"ID":"33498dd8-d969-4075-a1c7-479ee0ceaed4","Type":"ContainerStarted","Data":"0a06166855d124af9852c83c0e6d318ef0e9992a5acc7dcc31070bac843fb8e1"} Jul 10 19:28:31.102627 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:31.102596 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:31.121111 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:31.121060 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" podStartSLOduration=0.759925254 podStartE2EDuration="1.121045589s" totalImagesPullingTime="361.120335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:28:31.119481839 +0000 UTC m=+1687.870386715" watchObservedRunningTime="2026-07-10 19:28:31.121045589 +0000 UTC m=+1687.871950462" Jul 10 19:28:37.114202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:37.114171 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:28:44.154294 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:44.154253 2549 patch_prober.go:28] interesting pod/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:41142->10.133.0.55:15021: read: connection reset by peer" start-of-body= Jul 10 19:28:44.154697 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:44.154346 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:41142->10.133.0.55:15021: read: connection reset by peer" Jul 10 19:28:44.182589 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:44.182558 2549 generic.go:356] "Generic (PLEG): container finished" podID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerID="16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3" exitCode=0 Jul 10 19:28:44.182736 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:44.182629 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" event={"ID":"0b627fe5-3df1-4493-acb3-4e7f9bae83b2","Type":"ContainerDied","Data":"16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3"} Jul 10 19:28:44.183103 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:44.183084 2549 scope.go:117] "RemoveContainer" containerID="16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3" Jul 10 19:28:44.184284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:44.184249 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:44.184385 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:44.184326 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:44.184385 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:44.184366 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:28:45.189821 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:45.189720 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" event={"ID":"0b627fe5-3df1-4493-acb3-4e7f9bae83b2","Type":"ContainerStarted","Data":"e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635"} Jul 10 19:28:46.114745 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:46.114697 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:46.115000 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:46.114974 2549 patch_prober.go:28] interesting pod/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 19:28:46.115089 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:46.115038 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 19:28:47.115086 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:47.115047 2549 patch_prober.go:28] interesting pod/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 19:28:47.115574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:47.115120 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 19:28:48.114169 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:48.114132 2549 patch_prober.go:28] interesting pod/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 10 19:28:48.114431 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:48.114203 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 10 19:28:49.132990 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:49.132957 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:49.133358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:49.133305 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:28:49.134014 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:28:49.133998 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:29:17.352985 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.352948 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf"] Jul 10 19:29:17.353466 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.353232 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" podUID="33498dd8-d969-4075-a1c7-479ee0ceaed4" containerName="maas-api" containerID="cri-o://ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7" gracePeriod=30 Jul 10 19:29:17.609740 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.609679 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:29:17.740689 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.740653 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99b52\" (UniqueName: \"kubernetes.io/projected/33498dd8-d969-4075-a1c7-479ee0ceaed4-kube-api-access-99b52\") pod \"33498dd8-d969-4075-a1c7-479ee0ceaed4\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " Jul 10 19:29:17.740904 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.740716 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33498dd8-d969-4075-a1c7-479ee0ceaed4-openshift-service-ca\") pod \"33498dd8-d969-4075-a1c7-479ee0ceaed4\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " Jul 10 19:29:17.740904 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.740757 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33498dd8-d969-4075-a1c7-479ee0ceaed4-maas-api-tls\") pod \"33498dd8-d969-4075-a1c7-479ee0ceaed4\" (UID: \"33498dd8-d969-4075-a1c7-479ee0ceaed4\") " Jul 10 19:29:17.741157 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.741127 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33498dd8-d969-4075-a1c7-479ee0ceaed4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33498dd8-d969-4075-a1c7-479ee0ceaed4" (UID: "33498dd8-d969-4075-a1c7-479ee0ceaed4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:29:17.743020 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.742988 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33498dd8-d969-4075-a1c7-479ee0ceaed4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33498dd8-d969-4075-a1c7-479ee0ceaed4" (UID: "33498dd8-d969-4075-a1c7-479ee0ceaed4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:29:17.743131 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.743048 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33498dd8-d969-4075-a1c7-479ee0ceaed4-kube-api-access-99b52" (OuterVolumeSpecName: "kube-api-access-99b52") pod "33498dd8-d969-4075-a1c7-479ee0ceaed4" (UID: "33498dd8-d969-4075-a1c7-479ee0ceaed4"). InnerVolumeSpecName "kube-api-access-99b52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:29:17.842047 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.842010 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99b52\" (UniqueName: \"kubernetes.io/projected/33498dd8-d969-4075-a1c7-479ee0ceaed4-kube-api-access-99b52\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:29:17.842047 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.842038 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33498dd8-d969-4075-a1c7-479ee0ceaed4-openshift-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:29:17.842047 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:17.842049 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33498dd8-d969-4075-a1c7-479ee0ceaed4-maas-api-tls\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:29:18.359947 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.359916 2549 generic.go:356] "Generic (PLEG): container finished" podID="33498dd8-d969-4075-a1c7-479ee0ceaed4" containerID="ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7" exitCode=0 Jul 10 19:29:18.360354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.359950 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" event={"ID":"33498dd8-d969-4075-a1c7-479ee0ceaed4","Type":"ContainerDied","Data":"ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7"} Jul 10 19:29:18.360354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.359979 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" Jul 10 19:29:18.360354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.360026 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf" event={"ID":"33498dd8-d969-4075-a1c7-479ee0ceaed4","Type":"ContainerDied","Data":"0a06166855d124af9852c83c0e6d318ef0e9992a5acc7dcc31070bac843fb8e1"} Jul 10 19:29:18.360354 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.360061 2549 scope.go:117] "RemoveContainer" containerID="ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7" Jul 10 19:29:18.371853 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.371827 2549 scope.go:117] "RemoveContainer" containerID="ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7" Jul 10 19:29:18.372443 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:29:18.372417 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7\": container with ID starting with ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7 not found: ID does not exist" containerID="ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7" Jul 10 19:29:18.372508 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.372458 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7"} err="failed to get container status \"ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7\": rpc error: code = NotFound desc = could not find container \"ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7\": container with ID starting with ae53ecbf23bc29db9e459adfbdaec4a14f8d3b82de3f8de1952284be32fac4d7 not found: ID does not exist" Jul 10 19:29:18.377610 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.377583 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf"] Jul 10 19:29:18.379701 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:18.379678 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d2538db-8567ff4d-tbvdf"] Jul 10 19:29:19.804352 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:29:19.804307 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33498dd8-d969-4075-a1c7-479ee0ceaed4" path="/var/lib/kubelet/pods/33498dd8-d969-4075-a1c7-479ee0ceaed4/volumes" Jul 10 19:30:11.767410 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:11.767377 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv"] Jul 10 19:30:11.768042 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:11.767616 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" containerID="cri-o://e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635" gracePeriod=30 Jul 10 19:30:17.028843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.028746 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:30:17.188206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188174 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-podinfo\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188206 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188208 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-socket\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188249 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks27t\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-kube-api-access-ks27t\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188265 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istiod-ca-cert\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188287 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-credential-socket\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188310 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-token\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188326 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-certs\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188352 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-envoy\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188383 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-data\") pod \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\" (UID: \"0b627fe5-3df1-4493-acb3-4e7f9bae83b2\") " Jul 10 19:30:17.188770 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188622 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:30:17.188770 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188677 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:30:17.188934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188841 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.188934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188814 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:30:17.188934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188863 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.188934 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.188873 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:30:17.189651 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.189623 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:30:17.190598 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.190574 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:30:17.190787 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.190760 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:30:17.190787 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.190779 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-kube-api-access-ks27t" (OuterVolumeSpecName: "kube-api-access-ks27t") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "kube-api-access-ks27t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:30:17.190940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.190820 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b627fe5-3df1-4493-acb3-4e7f9bae83b2" (UID: "0b627fe5-3df1-4493-acb3-4e7f9bae83b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:30:17.289670 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.289601 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.289670 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.289624 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks27t\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-kube-api-access-ks27t\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.289670 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.289634 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.289670 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.289642 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.289670 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.289650 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.289670 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.289658 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.289670 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.289665 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b627fe5-3df1-4493-acb3-4e7f9bae83b2-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:30:17.652583 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.652548 2549 generic.go:356] "Generic (PLEG): container finished" podID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerID="e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635" exitCode=0 Jul 10 19:30:17.652747 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.652596 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" Jul 10 19:30:17.652747 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.652632 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" event={"ID":"0b627fe5-3df1-4493-acb3-4e7f9bae83b2","Type":"ContainerDied","Data":"e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635"} Jul 10 19:30:17.652747 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.652684 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv" event={"ID":"0b627fe5-3df1-4493-acb3-4e7f9bae83b2","Type":"ContainerDied","Data":"37c00faf24eac2c5ecb42b16dc1509ce1471a09d2ac986875c9083fd2677b1c2"} Jul 10 19:30:17.652747 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.652708 2549 scope.go:117] "RemoveContainer" containerID="e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635" Jul 10 19:30:17.662190 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.662173 2549 scope.go:117] "RemoveContainer" containerID="16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3" Jul 10 19:30:17.670138 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.670122 2549 scope.go:117] "RemoveContainer" containerID="e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635" Jul 10 19:30:17.670379 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:30:17.670362 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635\": container with ID starting with e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635 not found: ID does not exist" containerID="e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635" Jul 10 19:30:17.670414 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.670387 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635"} err="failed to get container status \"e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635\": rpc error: code = NotFound desc = could not find container \"e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635\": container with ID starting with e4b1d67b2581c41a77cb0bb138d22b842c756de977b9d2ac0a579d065257d635 not found: ID does not exist" Jul 10 19:30:17.670414 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.670402 2549 scope.go:117] "RemoveContainer" containerID="16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3" Jul 10 19:30:17.670623 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:30:17.670606 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3\": container with ID starting with 16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3 not found: ID does not exist" containerID="16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3" Jul 10 19:30:17.670659 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.670631 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3"} err="failed to get container status \"16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3\": rpc error: code = NotFound desc = could not find container \"16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3\": container with ID starting with 16101e92fb25b7b22f5b2af80d37ab818f8ac17d80d2f8a91d2c3b99c8ee2bc3 not found: ID does not exist" Jul 10 19:30:17.673830 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.673808 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv"] Jul 10 19:30:17.677441 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.677420 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d2538db-openshift-default-7d6b986fbc-kbbjv"] Jul 10 19:30:17.804613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:17.804581 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" path="/var/lib/kubelet/pods/0b627fe5-3df1-4493-acb3-4e7f9bae83b2/volumes" Jul 10 19:30:23.836405 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:23.836371 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:30:23.841770 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:23.841731 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:30:23.844600 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:23.844576 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:30:23.849123 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:30:23.849100 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:31:07.104831 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.104776 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v"] Jul 10 19:31:07.105284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105212 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" Jul 10 19:31:07.105284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105224 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" Jul 10 19:31:07.105284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105249 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" Jul 10 19:31:07.105284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105258 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" Jul 10 19:31:07.105284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105271 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33498dd8-d969-4075-a1c7-479ee0ceaed4" containerName="maas-api" Jul 10 19:31:07.105284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105277 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="33498dd8-d969-4075-a1c7-479ee0ceaed4" containerName="maas-api" Jul 10 19:31:07.105470 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105341 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" Jul 10 19:31:07.105470 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105349 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b627fe5-3df1-4493-acb3-4e7f9bae83b2" containerName="istio-proxy" Jul 10 19:31:07.105470 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.105357 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="33498dd8-d969-4075-a1c7-479ee0ceaed4" containerName="maas-api" Jul 10 19:31:07.108491 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.108475 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.122224 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.122205 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-40ce2b-openshift-default-dockercfg-nslkj\"" Jul 10 19:31:07.146803 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.146758 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v"] Jul 10 19:31:07.226127 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226090 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-credential-socket\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.226329 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226134 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-data\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.226329 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226152 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-podinfo\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.226329 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226232 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-certs\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.226329 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226270 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-socket\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.226329 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226289 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-envoy\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.226543 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226352 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6qh\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-kube-api-access-rn6qh\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.226543 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226381 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-token\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.226543 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.226437 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf0370-f5d0-437a-b9db-af5815d94f2d-istiod-ca-cert\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.327612 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327576 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-credential-socket\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.327843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327621 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-data\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.327843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327645 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-podinfo\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.327843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327681 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-certs\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.327843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327700 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-socket\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.327843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327726 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-envoy\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.327843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327757 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6qh\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-kube-api-access-rn6qh\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.327843 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327833 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-token\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.328216 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.327875 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf0370-f5d0-437a-b9db-af5815d94f2d-istiod-ca-cert\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.328216 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.328049 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-credential-socket\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.328216 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.328089 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-data\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.328376 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.328219 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-certs\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.328508 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.328467 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-socket\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.328779 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.328551 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf0370-f5d0-437a-b9db-af5815d94f2d-istiod-ca-cert\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.330437 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.330411 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-podinfo\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.330538 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.330524 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-envoy\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.342117 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.342094 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6qh\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-kube-api-access-rn6qh\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.344529 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.344507 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-token\") pod \"e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.418839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.418704 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:07.570130 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.570101 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v"] Jul 10 19:31:07.572671 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:31:07.572646 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5baf0370_f5d0_437a_b9db_af5815d94f2d.slice/crio-980ac4c46ccead86915251936689aeacaa22d978dfc3f2c12a960c91191e3b01 WatchSource:0}: Error finding container 980ac4c46ccead86915251936689aeacaa22d978dfc3f2c12a960c91191e3b01: Status 404 returned error can't find the container with id 980ac4c46ccead86915251936689aeacaa22d978dfc3f2c12a960c91191e3b01 Jul 10 19:31:07.574692 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.574662 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:07.574754 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.574740 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:07.574807 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.574771 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:07.889332 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.889294 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" event={"ID":"5baf0370-f5d0-437a-b9db-af5815d94f2d","Type":"ContainerStarted","Data":"1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c"} Jul 10 19:31:07.889579 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.889356 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" event={"ID":"5baf0370-f5d0-437a-b9db-af5815d94f2d","Type":"ContainerStarted","Data":"980ac4c46ccead86915251936689aeacaa22d978dfc3f2c12a960c91191e3b01"} Jul 10 19:31:07.911755 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:07.911697 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" podStartSLOduration=0.911675457 podStartE2EDuration="911.675457ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:07.91011933 +0000 UTC m=+1844.661024217" watchObservedRunningTime="2026-07-10 19:31:07.911675457 +0000 UTC m=+1844.662580333" Jul 10 19:31:08.419137 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:08.419091 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:08.424099 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:08.424074 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:08.894746 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:08.894713 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:08.895905 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:08.895882 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:10.046719 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.046682 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk"] Jul 10 19:31:10.050527 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.050508 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.053195 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.053175 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-40ce2b-serving-cert\"" Jul 10 19:31:10.062709 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.062686 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk"] Jul 10 19:31:10.158216 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.158182 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4131e09a-1fd8-4b8d-922c-211c893ad452-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.158559 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.158534 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhrbd\" (UniqueName: \"kubernetes.io/projected/4131e09a-1fd8-4b8d-922c-211c893ad452-kube-api-access-dhrbd\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.158732 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.158713 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4131e09a-1fd8-4b8d-922c-211c893ad452-maas-api-tls\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.259871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.259830 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4131e09a-1fd8-4b8d-922c-211c893ad452-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.260055 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.259893 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhrbd\" (UniqueName: \"kubernetes.io/projected/4131e09a-1fd8-4b8d-922c-211c893ad452-kube-api-access-dhrbd\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.260055 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.259926 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4131e09a-1fd8-4b8d-922c-211c893ad452-maas-api-tls\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.260530 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.260505 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4131e09a-1fd8-4b8d-922c-211c893ad452-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.262421 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.262403 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4131e09a-1fd8-4b8d-922c-211c893ad452-maas-api-tls\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.268685 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.268663 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhrbd\" (UniqueName: \"kubernetes.io/projected/4131e09a-1fd8-4b8d-922c-211c893ad452-kube-api-access-dhrbd\") pod \"maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.361476 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.361442 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:10.509421 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.509390 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk"] Jul 10 19:31:10.511338 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:31:10.511311 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4131e09a_1fd8_4b8d_922c_211c893ad452.slice/crio-dbb2cdbc79149c05918127e1f057eaf716678c970e87da9882d327b39eb8c3a3 WatchSource:0}: Error finding container dbb2cdbc79149c05918127e1f057eaf716678c970e87da9882d327b39eb8c3a3: Status 404 returned error can't find the container with id dbb2cdbc79149c05918127e1f057eaf716678c970e87da9882d327b39eb8c3a3 Jul 10 19:31:10.512760 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.512744 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:31:10.907193 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:10.907155 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" event={"ID":"4131e09a-1fd8-4b8d-922c-211c893ad452","Type":"ContainerStarted","Data":"dbb2cdbc79149c05918127e1f057eaf716678c970e87da9882d327b39eb8c3a3"} Jul 10 19:31:11.913587 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:11.913552 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" event={"ID":"4131e09a-1fd8-4b8d-922c-211c893ad452","Type":"ContainerStarted","Data":"729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd"} Jul 10 19:31:11.914005 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:11.913688 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:11.936431 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:11.936375 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" podStartSLOduration=1.4370558820000001 podStartE2EDuration="1.936356706s" totalImagesPullingTime="499.300824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:11.93374614 +0000 UTC m=+1848.684651045" watchObservedRunningTime="2026-07-10 19:31:11.936356706 +0000 UTC m=+1848.687261580" Jul 10 19:31:15.095242 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.095202 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp"] Jul 10 19:31:15.098983 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.098964 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.108900 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.108877 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-69b0c1-openshift-default-dockercfg-xxppr\"" Jul 10 19:31:15.127156 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.127124 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp"] Jul 10 19:31:15.210608 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210574 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-envoy\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.210807 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210639 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-certs\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.210807 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210676 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-credential-socket\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.210807 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210700 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-token\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.210807 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210734 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-socket\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.210990 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210820 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c8372-f988-4b27-bc85-3d31c45f700a-istiod-ca-cert\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.210990 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210854 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-data\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.210990 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210880 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-podinfo\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.210990 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.210940 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qq5q\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-kube-api-access-8qq5q\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311516 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311468 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-token\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311708 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311528 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-socket\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311708 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311554 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c8372-f988-4b27-bc85-3d31c45f700a-istiod-ca-cert\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311708 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311579 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-data\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311708 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311607 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-podinfo\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311743 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qq5q\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-kube-api-access-8qq5q\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311772 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-envoy\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311870 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-certs\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.311940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.311902 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-credential-socket\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.312163 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.312138 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-data\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.312230 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.312157 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-socket\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.312280 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.312251 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-credential-socket\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.312331 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.312283 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-certs\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.312524 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.312501 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c8372-f988-4b27-bc85-3d31c45f700a-istiod-ca-cert\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.314409 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.314389 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-podinfo\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.314409 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.314400 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-envoy\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.325880 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.325854 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qq5q\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-kube-api-access-8qq5q\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.336936 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.336911 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-token\") pod \"e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.409290 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.409188 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:15.570536 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.570507 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp"] Jul 10 19:31:15.572677 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:31:15.572645 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4c8372_f988_4b27_bc85_3d31c45f700a.slice/crio-ef93455c7ce7ebcad6bfde54e64bc3dd76e8449166d3501d584ffb76ca0019b2 WatchSource:0}: Error finding container ef93455c7ce7ebcad6bfde54e64bc3dd76e8449166d3501d584ffb76ca0019b2: Status 404 returned error can't find the container with id ef93455c7ce7ebcad6bfde54e64bc3dd76e8449166d3501d584ffb76ca0019b2 Jul 10 19:31:15.574983 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.574950 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:15.575042 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.575017 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:15.575082 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.575048 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:15.934629 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.934587 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" event={"ID":"6e4c8372-f988-4b27-bc85-3d31c45f700a","Type":"ContainerStarted","Data":"fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835"} Jul 10 19:31:15.934814 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.934658 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" event={"ID":"6e4c8372-f988-4b27-bc85-3d31c45f700a","Type":"ContainerStarted","Data":"ef93455c7ce7ebcad6bfde54e64bc3dd76e8449166d3501d584ffb76ca0019b2"} Jul 10 19:31:15.963481 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:15.963422 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" podStartSLOduration=0.963405529 podStartE2EDuration="963.405529ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:15.96051553 +0000 UTC m=+1852.711420430" watchObservedRunningTime="2026-07-10 19:31:15.963405529 +0000 UTC m=+1852.714310402" Jul 10 19:31:16.409976 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:16.409937 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:16.415188 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:16.415162 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:16.941265 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:16.941231 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:16.942214 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:16.942195 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:17.924582 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:17.924550 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:31:24.937035 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:24.936990 2549 patch_prober.go:28] interesting pod/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 19:31:24.937570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:24.937060 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 19:31:26.996383 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:26.996347 2549 generic.go:356] "Generic (PLEG): container finished" podID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerID="1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c" exitCode=0 Jul 10 19:31:26.996830 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:26.996432 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" event={"ID":"5baf0370-f5d0-437a-b9db-af5815d94f2d","Type":"ContainerDied","Data":"1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c"} Jul 10 19:31:26.996897 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:26.996882 2549 scope.go:117] "RemoveContainer" containerID="1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c" Jul 10 19:31:26.997919 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:26.997892 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:26.997978 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:26.997962 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:26.998017 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:26.998002 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:28.003812 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:28.003748 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" event={"ID":"5baf0370-f5d0-437a-b9db-af5815d94f2d","Type":"ContainerStarted","Data":"c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277"} Jul 10 19:31:28.419730 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:28.419690 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:28.419950 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:28.419828 2549 patch_prober.go:28] interesting pod/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 19:31:28.419950 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:28.419896 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 19:31:29.419381 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:29.419323 2549 patch_prober.go:28] interesting pod/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 19:31:29.419821 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:29.419400 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 19:31:30.419631 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:30.419590 2549 patch_prober.go:28] interesting pod/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 19:31:30.420057 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:30.419646 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 19:31:31.419628 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:31.419592 2549 patch_prober.go:28] interesting pod/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 10 19:31:31.419844 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:31.419644 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 10 19:31:32.422883 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:32.422849 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:32.423257 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:32.423128 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:32.423972 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:32.423954 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:31:35.044157 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:35.044127 2549 generic.go:356] "Generic (PLEG): container finished" podID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerID="fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835" exitCode=0 Jul 10 19:31:35.044672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:35.044225 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" event={"ID":"6e4c8372-f988-4b27-bc85-3d31c45f700a","Type":"ContainerDied","Data":"fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835"} Jul 10 19:31:35.044976 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:35.044953 2549 scope.go:117] "RemoveContainer" containerID="fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835" Jul 10 19:31:35.046221 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:35.046192 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:35.046297 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:35.046280 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:35.046336 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:35.046326 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:31:36.051859 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:36.051821 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" event={"ID":"6e4c8372-f988-4b27-bc85-3d31c45f700a","Type":"ContainerStarted","Data":"9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863"} Jul 10 19:31:36.409957 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:36.409921 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:36.410196 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:36.410175 2549 patch_prober.go:28] interesting pod/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 19:31:36.410242 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:36.410224 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 19:31:37.409886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:37.409847 2549 patch_prober.go:28] interesting pod/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 19:31:37.410387 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:37.409915 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 19:31:38.409606 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:38.409568 2549 patch_prober.go:28] interesting pod/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 19:31:38.409909 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:38.409626 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 19:31:39.410320 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:39.410281 2549 patch_prober.go:28] interesting pod/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 10 19:31:39.410717 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:39.410353 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 10 19:31:40.426237 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:40.426201 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:40.426713 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:40.426651 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:31:40.427903 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:31:40.427878 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:32:27.793710 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:27.793675 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk"] Jul 10 19:32:27.794298 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:27.793925 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" podUID="4131e09a-1fd8-4b8d-922c-211c893ad452" containerName="maas-api" containerID="cri-o://729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd" gracePeriod=30 Jul 10 19:32:27.920270 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:27.920209 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" podUID="4131e09a-1fd8-4b8d-922c-211c893ad452" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.58:8443/health\": dial tcp 10.133.0.58:8443: connect: connection refused" Jul 10 19:32:28.246114 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.246087 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:32:28.318325 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.318293 2549 generic.go:356] "Generic (PLEG): container finished" podID="4131e09a-1fd8-4b8d-922c-211c893ad452" containerID="729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd" exitCode=0 Jul 10 19:32:28.318518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.318340 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" Jul 10 19:32:28.318518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.318344 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" event={"ID":"4131e09a-1fd8-4b8d-922c-211c893ad452","Type":"ContainerDied","Data":"729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd"} Jul 10 19:32:28.318518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.318463 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk" event={"ID":"4131e09a-1fd8-4b8d-922c-211c893ad452","Type":"ContainerDied","Data":"dbb2cdbc79149c05918127e1f057eaf716678c970e87da9882d327b39eb8c3a3"} Jul 10 19:32:28.318518 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.318486 2549 scope.go:117] "RemoveContainer" containerID="729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd" Jul 10 19:32:28.327353 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.327332 2549 scope.go:117] "RemoveContainer" containerID="729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd" Jul 10 19:32:28.327674 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:32:28.327650 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd\": container with ID starting with 729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd not found: ID does not exist" containerID="729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd" Jul 10 19:32:28.327772 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.327682 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd"} err="failed to get container status \"729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd\": rpc error: code = NotFound desc = could not find container \"729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd\": container with ID starting with 729a021b102fcb29eb28fcedbfa969ae96ba5de3fbdc4596e15e250e221359dd not found: ID does not exist" Jul 10 19:32:28.342130 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.342102 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhrbd\" (UniqueName: \"kubernetes.io/projected/4131e09a-1fd8-4b8d-922c-211c893ad452-kube-api-access-dhrbd\") pod \"4131e09a-1fd8-4b8d-922c-211c893ad452\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " Jul 10 19:32:28.342304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.342218 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4131e09a-1fd8-4b8d-922c-211c893ad452-maas-api-tls\") pod \"4131e09a-1fd8-4b8d-922c-211c893ad452\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " Jul 10 19:32:28.342304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.342277 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4131e09a-1fd8-4b8d-922c-211c893ad452-openshift-service-ca\") pod \"4131e09a-1fd8-4b8d-922c-211c893ad452\" (UID: \"4131e09a-1fd8-4b8d-922c-211c893ad452\") " Jul 10 19:32:28.342624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.342595 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4131e09a-1fd8-4b8d-922c-211c893ad452-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4131e09a-1fd8-4b8d-922c-211c893ad452" (UID: "4131e09a-1fd8-4b8d-922c-211c893ad452"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:32:28.344525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.344492 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4131e09a-1fd8-4b8d-922c-211c893ad452-kube-api-access-dhrbd" (OuterVolumeSpecName: "kube-api-access-dhrbd") pod "4131e09a-1fd8-4b8d-922c-211c893ad452" (UID: "4131e09a-1fd8-4b8d-922c-211c893ad452"). InnerVolumeSpecName "kube-api-access-dhrbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:32:28.344624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.344527 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4131e09a-1fd8-4b8d-922c-211c893ad452-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4131e09a-1fd8-4b8d-922c-211c893ad452" (UID: "4131e09a-1fd8-4b8d-922c-211c893ad452"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:32:28.444135 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.444097 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4131e09a-1fd8-4b8d-922c-211c893ad452-maas-api-tls\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:32:28.444135 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.444128 2549 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4131e09a-1fd8-4b8d-922c-211c893ad452-openshift-service-ca\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:32:28.444135 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.444138 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhrbd\" (UniqueName: \"kubernetes.io/projected/4131e09a-1fd8-4b8d-922c-211c893ad452-kube-api-access-dhrbd\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:32:28.641039 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.641010 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk"] Jul 10 19:32:28.646322 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:28.646294 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40ce2b-67c7f84bff-llktk"] Jul 10 19:32:29.805723 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:32:29.805686 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4131e09a-1fd8-4b8d-922c-211c893ad452" path="/var/lib/kubelet/pods/4131e09a-1fd8-4b8d-922c-211c893ad452/volumes" Jul 10 19:33:10.034580 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:10.034538 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v"] Jul 10 19:33:10.035534 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:10.035498 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" containerID="cri-o://c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277" gracePeriod=30 Jul 10 19:33:15.303757 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.303687 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:33:15.394295 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394253 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-certs\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.394491 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394337 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn6qh\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-kube-api-access-rn6qh\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.394491 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394380 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-token\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.394491 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394402 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-envoy\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.394491 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394429 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-credential-socket\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.394491 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394458 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-socket\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.394752 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394516 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-podinfo\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.394752 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394514 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:15.394752 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394541 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-data\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.394752 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394612 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf0370-f5d0-437a-b9db-af5815d94f2d-istiod-ca-cert\") pod \"5baf0370-f5d0-437a-b9db-af5815d94f2d\" (UID: \"5baf0370-f5d0-437a-b9db-af5815d94f2d\") " Jul 10 19:33:15.395008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394781 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:15.395008 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.394827 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:15.395118 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.395078 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baf0370-f5d0-437a-b9db-af5815d94f2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:33:15.395223 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.395204 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.395286 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.395229 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.395286 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.395245 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baf0370-f5d0-437a-b9db-af5815d94f2d-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.395286 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.395259 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.395574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.395555 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:15.397241 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.397214 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:15.397306 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.397240 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:15.397367 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.397348 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-kube-api-access-rn6qh" (OuterVolumeSpecName: "kube-api-access-rn6qh") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "kube-api-access-rn6qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:15.397439 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.397417 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5baf0370-f5d0-437a-b9db-af5815d94f2d" (UID: "5baf0370-f5d0-437a-b9db-af5815d94f2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:33:15.496281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.496237 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.496281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.496274 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.496281 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.496287 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6qh\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-kube-api-access-rn6qh\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.496525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.496300 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.496525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.496312 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baf0370-f5d0-437a-b9db-af5815d94f2d-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:15.550102 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.550068 2549 generic.go:356] "Generic (PLEG): container finished" podID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerID="c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277" exitCode=0 Jul 10 19:33:15.550269 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.550120 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" Jul 10 19:33:15.550269 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.550161 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" event={"ID":"5baf0370-f5d0-437a-b9db-af5815d94f2d","Type":"ContainerDied","Data":"c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277"} Jul 10 19:33:15.550269 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.550218 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v" event={"ID":"5baf0370-f5d0-437a-b9db-af5815d94f2d","Type":"ContainerDied","Data":"980ac4c46ccead86915251936689aeacaa22d978dfc3f2c12a960c91191e3b01"} Jul 10 19:33:15.550269 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.550251 2549 scope.go:117] "RemoveContainer" containerID="c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277" Jul 10 19:33:15.559826 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.559784 2549 scope.go:117] "RemoveContainer" containerID="1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c" Jul 10 19:33:15.568114 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.568095 2549 scope.go:117] "RemoveContainer" containerID="c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277" Jul 10 19:33:15.568416 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:33:15.568389 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277\": container with ID starting with c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277 not found: ID does not exist" containerID="c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277" Jul 10 19:33:15.568526 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.568423 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277"} err="failed to get container status \"c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277\": rpc error: code = NotFound desc = could not find container \"c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277\": container with ID starting with c3654065e884b10b761acf4eb5f76ab55493a8d3bd40f1552f6df17cba59d277 not found: ID does not exist" Jul 10 19:33:15.568526 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.568441 2549 scope.go:117] "RemoveContainer" containerID="1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c" Jul 10 19:33:15.569063 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:33:15.569035 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c\": container with ID starting with 1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c not found: ID does not exist" containerID="1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c" Jul 10 19:33:15.569162 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.569071 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c"} err="failed to get container status \"1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c\": rpc error: code = NotFound desc = could not find container \"1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c\": container with ID starting with 1b5b10400cd4cffbc98c96cf935f570378b5eb6fde4807530c7343811d069e5c not found: ID does not exist" Jul 10 19:33:15.571133 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.571113 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v"] Jul 10 19:33:15.575031 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.575009 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-40ce2b-openshift-default-5f46d8c8b9-f722v"] Jul 10 19:33:15.805164 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:15.805123 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" path="/var/lib/kubelet/pods/5baf0370-f5d0-437a-b9db-af5815d94f2d/volumes" Jul 10 19:33:17.021392 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.021359 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp"] Jul 10 19:33:17.021767 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.021600 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" containerID="cri-o://9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863" gracePeriod=30 Jul 10 19:33:17.830899 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.830863 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx"] Jul 10 19:33:17.831288 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.831271 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4131e09a-1fd8-4b8d-922c-211c893ad452" containerName="maas-api" Jul 10 19:33:17.831288 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.831288 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4131e09a-1fd8-4b8d-922c-211c893ad452" containerName="maas-api" Jul 10 19:33:17.831430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.831303 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" Jul 10 19:33:17.831430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.831310 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" Jul 10 19:33:17.831430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.831377 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="4131e09a-1fd8-4b8d-922c-211c893ad452" containerName="maas-api" Jul 10 19:33:17.831430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.831385 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" Jul 10 19:33:17.831430 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.831394 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" Jul 10 19:33:17.836155 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.836125 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.838705 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.838681 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-6669d3-openshift-default-dockercfg-2nm6w\"" Jul 10 19:33:17.848901 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.848878 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx"] Jul 10 19:33:17.921871 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.921830 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-credential-socket\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.922094 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.921879 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-socket\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.922094 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.921937 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49bb4952-6416-403d-ab19-75ae6c4e7209-istiod-ca-cert\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.922094 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.921968 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-certs\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.922094 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.922048 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-podinfo\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.922312 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.922118 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-token\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.922312 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.922216 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-envoy\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.922312 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.922243 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkwt\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-kube-api-access-hdkwt\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:17.922474 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:17.922381 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-data\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023203 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023164 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-token\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023203 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023211 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-envoy\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023228 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkwt\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-kube-api-access-hdkwt\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023284 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-data\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023311 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-credential-socket\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023329 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-socket\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023364 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49bb4952-6416-403d-ab19-75ae6c4e7209-istiod-ca-cert\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023395 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-certs\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.023748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023420 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-podinfo\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.024309 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023751 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-data\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.024309 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023880 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-certs\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.024309 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023963 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-socket\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.024309 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.023976 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-credential-socket\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.024309 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.024118 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49bb4952-6416-403d-ab19-75ae6c4e7209-istiod-ca-cert\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.025784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.025756 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-envoy\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.026267 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.026245 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-podinfo\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.032189 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.032166 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-token\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.032284 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.032241 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkwt\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-kube-api-access-hdkwt\") pod \"e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.148743 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.148648 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:18.297483 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.297455 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx"] Jul 10 19:33:18.300218 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:33:18.300193 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49bb4952_6416_403d_ab19_75ae6c4e7209.slice/crio-657c6bf5cc6539efab8db7b8e6bc63176c2fa5ea5371a873c0e10826e6d64590 WatchSource:0}: Error finding container 657c6bf5cc6539efab8db7b8e6bc63176c2fa5ea5371a873c0e10826e6d64590: Status 404 returned error can't find the container with id 657c6bf5cc6539efab8db7b8e6bc63176c2fa5ea5371a873c0e10826e6d64590 Jul 10 19:33:18.302393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.302362 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:18.302458 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.302427 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:18.302458 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.302455 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:18.572183 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.572140 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" event={"ID":"49bb4952-6416-403d-ab19-75ae6c4e7209","Type":"ContainerStarted","Data":"fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e"} Jul 10 19:33:18.573296 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.573266 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" event={"ID":"49bb4952-6416-403d-ab19-75ae6c4e7209","Type":"ContainerStarted","Data":"657c6bf5cc6539efab8db7b8e6bc63176c2fa5ea5371a873c0e10826e6d64590"} Jul 10 19:33:18.596666 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:18.596606 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" podStartSLOduration=1.596590789 podStartE2EDuration="1.596590789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:33:18.593951392 +0000 UTC m=+1975.344856279" watchObservedRunningTime="2026-07-10 19:33:18.596590789 +0000 UTC m=+1975.347495663" Jul 10 19:33:19.148975 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:19.148938 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:19.154146 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:19.154118 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:19.577061 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:19.577030 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:19.578145 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:19.578126 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:22.291981 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.291912 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:33:22.365401 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365369 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-data\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365604 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365436 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-envoy\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365604 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365478 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-token\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365604 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365507 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-podinfo\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365604 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365529 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qq5q\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-kube-api-access-8qq5q\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365604 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365547 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-socket\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365641 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-certs\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365689 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c8372-f988-4b27-bc85-3d31c45f700a-istiod-ca-cert\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365724 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-credential-socket\") pod \"6e4c8372-f988-4b27-bc85-3d31c45f700a\" (UID: \"6e4c8372-f988-4b27-bc85-3d31c45f700a\") " Jul 10 19:33:22.365930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.365863 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:22.366412 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.366134 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.366412 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.366344 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:22.366654 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.366556 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:22.366654 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.366577 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:22.366654 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.366583 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4c8372-f988-4b27-bc85-3d31c45f700a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:33:22.367930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.367907 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:22.368077 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.368059 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:33:22.368373 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.368354 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-kube-api-access-8qq5q" (OuterVolumeSpecName: "kube-api-access-8qq5q") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "kube-api-access-8qq5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:22.368459 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.368443 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e4c8372-f988-4b27-bc85-3d31c45f700a" (UID: "6e4c8372-f988-4b27-bc85-3d31c45f700a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:22.466839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.466773 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.466839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.466831 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.466839 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.466846 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.467100 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.466858 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qq5q\" (UniqueName: \"kubernetes.io/projected/6e4c8372-f988-4b27-bc85-3d31c45f700a-kube-api-access-8qq5q\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.467100 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.466870 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.467100 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.466882 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c8372-f988-4b27-bc85-3d31c45f700a-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.467100 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.466892 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.467100 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.466902 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c8372-f988-4b27-bc85-3d31c45f700a-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:22.592930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.592899 2549 generic.go:356] "Generic (PLEG): container finished" podID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerID="9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863" exitCode=0 Jul 10 19:33:22.593101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.592947 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" Jul 10 19:33:22.593101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.592966 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" event={"ID":"6e4c8372-f988-4b27-bc85-3d31c45f700a","Type":"ContainerDied","Data":"9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863"} Jul 10 19:33:22.593101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.593015 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp" event={"ID":"6e4c8372-f988-4b27-bc85-3d31c45f700a","Type":"ContainerDied","Data":"ef93455c7ce7ebcad6bfde54e64bc3dd76e8449166d3501d584ffb76ca0019b2"} Jul 10 19:33:22.593101 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.593036 2549 scope.go:117] "RemoveContainer" containerID="9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863" Jul 10 19:33:22.603111 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.603091 2549 scope.go:117] "RemoveContainer" containerID="fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835" Jul 10 19:33:22.612075 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.612040 2549 scope.go:117] "RemoveContainer" containerID="9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863" Jul 10 19:33:22.612767 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:33:22.612742 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863\": container with ID starting with 9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863 not found: ID does not exist" containerID="9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863" Jul 10 19:33:22.612960 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.612775 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863"} err="failed to get container status \"9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863\": rpc error: code = NotFound desc = could not find container \"9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863\": container with ID starting with 9c7fd25eb01abf17681563cbd7e76c79682eb40432e8904a2b142cd657de2863 not found: ID does not exist" Jul 10 19:33:22.612960 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.612826 2549 scope.go:117] "RemoveContainer" containerID="fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835" Jul 10 19:33:22.613379 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:33:22.613354 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835\": container with ID starting with fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835 not found: ID does not exist" containerID="fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835" Jul 10 19:33:22.613479 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.613389 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835"} err="failed to get container status \"fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835\": rpc error: code = NotFound desc = could not find container \"fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835\": container with ID starting with fad9e830b1ca1820c3f8187f1508daa3f21edcaaf4eb65f06821185b5f9c5835 not found: ID does not exist" Jul 10 19:33:22.615591 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.615565 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp"] Jul 10 19:33:22.619098 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:22.619068 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-69b0c1-openshift-default-7df95b78b9-dg6zp"] Jul 10 19:33:23.809000 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:23.807073 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" path="/var/lib/kubelet/pods/6e4c8372-f988-4b27-bc85-3d31c45f700a/volumes" Jul 10 19:33:25.589072 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589035 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf"] Jul 10 19:33:25.589630 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589612 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" Jul 10 19:33:25.589673 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589633 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5baf0370-f5d0-437a-b9db-af5815d94f2d" containerName="istio-proxy" Jul 10 19:33:25.589673 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589652 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" Jul 10 19:33:25.589673 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589661 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" Jul 10 19:33:25.589776 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589684 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" Jul 10 19:33:25.589776 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589693 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" Jul 10 19:33:25.589905 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589805 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" Jul 10 19:33:25.589905 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.589820 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4c8372-f988-4b27-bc85-3d31c45f700a" containerName="istio-proxy" Jul 10 19:33:25.594883 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.594862 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.597557 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.597533 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-094d6e-openshift-default-dockercfg-k6wkk\"" Jul 10 19:33:25.602545 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.602520 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf"] Jul 10 19:33:25.698913 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.698862 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-data\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.699093 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.698928 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-envoy\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.699093 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.698957 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-podinfo\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.699093 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.698981 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-token\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.699093 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.699028 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-certs\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.699093 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.699062 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-credential-socket\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.699093 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.699085 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9m6b\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-kube-api-access-r9m6b\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.699411 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.699111 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-socket\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.699411 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.699139 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istiod-ca-cert\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.800460 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800421 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-data\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.800643 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800487 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-envoy\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.800643 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800518 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-podinfo\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.800643 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800539 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-token\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.800643 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800580 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-certs\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.800643 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800611 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-credential-socket\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.800643 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800633 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9m6b\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-kube-api-access-r9m6b\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.801146 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800659 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-socket\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.801146 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800687 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istiod-ca-cert\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.801146 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.800925 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-data\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.801146 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.801068 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-credential-socket\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.801370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.801230 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-socket\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.801370 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.801302 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-certs\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.801496 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.801472 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istiod-ca-cert\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.803243 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.803220 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-envoy\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.803331 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.803262 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-podinfo\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.808394 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.808363 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-token\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.808511 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.808493 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9m6b\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-kube-api-access-r9m6b\") pod \"e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:25.905729 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:25.905622 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:26.048053 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:26.048022 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf"] Jul 10 19:33:26.051179 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:33:26.051152 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ade91d7_54dd_46bd_9f14_c0c2d44a665f.slice/crio-8443de76be9fb666bf038d9483b7a75663c50720615f289f5c6d81d2b6ad2030 WatchSource:0}: Error finding container 8443de76be9fb666bf038d9483b7a75663c50720615f289f5c6d81d2b6ad2030: Status 404 returned error can't find the container with id 8443de76be9fb666bf038d9483b7a75663c50720615f289f5c6d81d2b6ad2030 Jul 10 19:33:26.056369 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:26.056325 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:26.056478 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:26.056421 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:26.056478 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:26.056463 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:26.615589 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:26.615551 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" event={"ID":"1ade91d7-54dd-46bd-9f14-c0c2d44a665f","Type":"ContainerStarted","Data":"8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd"} Jul 10 19:33:26.616027 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:26.615608 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" event={"ID":"1ade91d7-54dd-46bd-9f14-c0c2d44a665f","Type":"ContainerStarted","Data":"8443de76be9fb666bf038d9483b7a75663c50720615f289f5c6d81d2b6ad2030"} Jul 10 19:33:26.638876 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:26.638816 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" podStartSLOduration=1.638780192 podStartE2EDuration="1.638780192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:33:26.637782173 +0000 UTC m=+1983.388687048" watchObservedRunningTime="2026-07-10 19:33:26.638780192 +0000 UTC m=+1983.389685066" Jul 10 19:33:26.906664 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:26.906577 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:27.911818 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:27.911764 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:28.625834 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:28.625787 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:28.626805 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:28.626770 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:35.620006 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:35.619951 2549 patch_prober.go:28] interesting pod/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 19:33:35.620515 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:35.620033 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 19:33:36.677525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:36.677433 2549 generic.go:356] "Generic (PLEG): container finished" podID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerID="fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e" exitCode=0 Jul 10 19:33:36.677525 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:36.677482 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" event={"ID":"49bb4952-6416-403d-ab19-75ae6c4e7209","Type":"ContainerDied","Data":"fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e"} Jul 10 19:33:36.678030 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:36.677971 2549 scope.go:117] "RemoveContainer" containerID="fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e" Jul 10 19:33:36.678969 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:36.678937 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:36.679051 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:36.679014 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:36.679095 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:36.679050 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:37.684348 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:37.684314 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" event={"ID":"49bb4952-6416-403d-ab19-75ae6c4e7209","Type":"ContainerStarted","Data":"aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae"} Jul 10 19:33:38.149233 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:38.149189 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:38.149445 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:38.149329 2549 patch_prober.go:28] interesting pod/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 19:33:38.149445 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:38.149373 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 19:33:39.149810 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:39.149749 2549 patch_prober.go:28] interesting pod/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 19:33:39.150337 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:39.149846 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 19:33:40.149566 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:40.149513 2549 patch_prober.go:28] interesting pod/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 19:33:40.149951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:40.149916 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 19:33:41.149896 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:41.149846 2549 patch_prober.go:28] interesting pod/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 10 19:33:41.150211 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:41.149919 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 10 19:33:42.158379 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:42.158337 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:42.158898 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:42.158734 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:42.159574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:42.159555 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:44.668019 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:44.667968 2549 patch_prober.go:28] interesting pod/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 19:33:44.668408 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:44.668032 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 19:33:46.734482 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:46.734448 2549 generic.go:356] "Generic (PLEG): container finished" podID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerID="8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd" exitCode=0 Jul 10 19:33:46.734925 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:46.734529 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" event={"ID":"1ade91d7-54dd-46bd-9f14-c0c2d44a665f","Type":"ContainerDied","Data":"8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd"} Jul 10 19:33:46.734984 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:46.734961 2549 scope.go:117] "RemoveContainer" containerID="8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd" Jul 10 19:33:46.736102 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:46.736072 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:46.736208 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:46.736159 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:46.736272 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:46.736205 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:47.745620 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:47.745573 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" event={"ID":"1ade91d7-54dd-46bd-9f14-c0c2d44a665f","Type":"ContainerStarted","Data":"8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63"} Jul 10 19:33:47.906327 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:47.906286 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:47.906517 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:47.906497 2549 patch_prober.go:28] interesting pod/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 10 19:33:47.906570 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:47.906548 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 10 19:33:48.025347 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:48.025264 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx"] Jul 10 19:33:48.025497 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:48.025476 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" containerID="cri-o://aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae" gracePeriod=30 Jul 10 19:33:48.906972 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:48.906940 2549 patch_prober.go:28] interesting pod/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 10 19:33:48.907378 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:48.906992 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 10 19:33:49.906110 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:49.906069 2549 patch_prober.go:28] interesting pod/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 10 19:33:49.906295 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:49.906126 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 10 19:33:50.906613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:50.906576 2549 patch_prober.go:28] interesting pod/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 10 19:33:50.907133 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:50.906636 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 10 19:33:51.909867 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:51.909836 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:51.910227 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:51.910102 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:51.910786 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:51.910771 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:33:53.193868 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.193840 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:53.290446 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290347 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-data\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.290446 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290391 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-envoy\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.290446 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290433 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49bb4952-6416-403d-ab19-75ae6c4e7209-istiod-ca-cert\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.290743 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290470 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-socket\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.290743 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290503 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-token\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.290743 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290527 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-podinfo\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.290743 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290554 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-credential-socket\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.290743 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290581 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-certs\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.290743 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290690 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdkwt\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-kube-api-access-hdkwt\") pod \"49bb4952-6416-403d-ab19-75ae6c4e7209\" (UID: \"49bb4952-6416-403d-ab19-75ae6c4e7209\") " Jul 10 19:33:53.291091 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290820 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:53.291091 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.290930 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49bb4952-6416-403d-ab19-75ae6c4e7209-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:33:53.291223 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.291133 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:53.291223 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.291150 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:53.291328 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.291209 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49bb4952-6416-403d-ab19-75ae6c4e7209-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.291328 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.291242 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.291612 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.291547 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-data" (OuterVolumeSpecName: "istio-data") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:53.292955 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.292930 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-token" (OuterVolumeSpecName: "istio-token") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:53.293077 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.293030 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:33:53.293134 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.293089 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:33:53.293134 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.293094 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-kube-api-access-hdkwt" (OuterVolumeSpecName: "kube-api-access-hdkwt") pod "49bb4952-6416-403d-ab19-75ae6c4e7209" (UID: "49bb4952-6416-403d-ab19-75ae6c4e7209"). InnerVolumeSpecName "kube-api-access-hdkwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:33:53.392019 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.391985 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkwt\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-kube-api-access-hdkwt\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.392019 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.392016 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.392019 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.392026 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.392254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.392033 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.392254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.392043 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49bb4952-6416-403d-ab19-75ae6c4e7209-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.392254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.392054 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.392254 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.392062 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49bb4952-6416-403d-ab19-75ae6c4e7209-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:33:53.780991 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.780955 2549 generic.go:356] "Generic (PLEG): container finished" podID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerID="aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae" exitCode=0 Jul 10 19:33:53.781202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.781006 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" Jul 10 19:33:53.781202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.781051 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" event={"ID":"49bb4952-6416-403d-ab19-75ae6c4e7209","Type":"ContainerDied","Data":"aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae"} Jul 10 19:33:53.781202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.781104 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx" event={"ID":"49bb4952-6416-403d-ab19-75ae6c4e7209","Type":"ContainerDied","Data":"657c6bf5cc6539efab8db7b8e6bc63176c2fa5ea5371a873c0e10826e6d64590"} Jul 10 19:33:53.781202 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.781126 2549 scope.go:117] "RemoveContainer" containerID="aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae" Jul 10 19:33:53.791532 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.791510 2549 scope.go:117] "RemoveContainer" containerID="fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e" Jul 10 19:33:53.799820 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.799771 2549 scope.go:117] "RemoveContainer" containerID="aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae" Jul 10 19:33:53.800116 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:33:53.800090 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae\": container with ID starting with aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae not found: ID does not exist" containerID="aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae" Jul 10 19:33:53.800225 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.800129 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae"} err="failed to get container status \"aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae\": rpc error: code = NotFound desc = could not find container \"aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae\": container with ID starting with aff165a8123030c263647f03a2db94e68ea6aa42eac3e7dec154448ac4dc32ae not found: ID does not exist" Jul 10 19:33:53.800225 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.800154 2549 scope.go:117] "RemoveContainer" containerID="fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e" Jul 10 19:33:53.800564 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:33:53.800526 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e\": container with ID starting with fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e not found: ID does not exist" containerID="fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e" Jul 10 19:33:53.800674 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.800564 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e"} err="failed to get container status \"fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e\": rpc error: code = NotFound desc = could not find container \"fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e\": container with ID starting with fff8debfdcb44ec2421f1c779c08c60c623fb3b8628570789844979c55ad6b3e not found: ID does not exist" Jul 10 19:33:53.807312 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.807292 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx"] Jul 10 19:33:53.807410 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:53.807321 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-6669d3-openshift-default-5554d96747-w7kjx"] Jul 10 19:33:54.908303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:54.908270 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf"] Jul 10 19:33:54.908787 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:54.908465 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" containerID="cri-o://8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63" gracePeriod=30 Jul 10 19:33:55.742093 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.742054 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk"] Jul 10 19:33:55.742677 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.742653 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" Jul 10 19:33:55.742677 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.742675 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" Jul 10 19:33:55.742856 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.742770 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" Jul 10 19:33:55.742856 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.742778 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" Jul 10 19:33:55.745440 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.745394 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.747784 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.747764 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b1f450-openshift-default-dockercfg-hclsz\"" Jul 10 19:33:55.755924 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.755899 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk"] Jul 10 19:33:55.805304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.805274 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" path="/var/lib/kubelet/pods/49bb4952-6416-403d-ab19-75ae6c4e7209/volumes" Jul 10 19:33:55.816535 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816505 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jts\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-kube-api-access-z6jts\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.816660 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816546 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-credential-socket\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.816660 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816608 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-istio-token\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.816660 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816642 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197355fb-227f-473d-abea-bbcec07103b6-istio-podinfo\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.816819 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816671 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197355fb-227f-473d-abea-bbcec07103b6-istiod-ca-cert\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.816819 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816715 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-envoy\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.816819 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816739 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-certs\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.816819 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816784 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-data\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.816955 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.816834 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-socket\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.917930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.917893 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-data\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.917930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.917932 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-socket\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918063 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jts\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-kube-api-access-z6jts\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918113 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-credential-socket\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918177 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-istio-token\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918208 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197355fb-227f-473d-abea-bbcec07103b6-istio-podinfo\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918253 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197355fb-227f-473d-abea-bbcec07103b6-istiod-ca-cert\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918285 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-socket\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918301 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-envoy\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918393 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918328 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-certs\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918771 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918419 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-data\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918771 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918428 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-credential-socket\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918771 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918650 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-certs\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.918904 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.918864 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197355fb-227f-473d-abea-bbcec07103b6-istiod-ca-cert\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.920878 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.920859 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-envoy\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.920939 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.920871 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197355fb-227f-473d-abea-bbcec07103b6-istio-podinfo\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.933290 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.933263 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-istio-token\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:55.933847 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:55.933827 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jts\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-kube-api-access-z6jts\") pod \"e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:56.057185 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:56.057095 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:56.201617 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:56.201583 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk"] Jul 10 19:33:56.204572 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:33:56.204539 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197355fb_227f_473d_abea_bbcec07103b6.slice/crio-84484be701249cbe587c78da6edd556c2367e87a5c4230f06b0b4829dc75066d WatchSource:0}: Error finding container 84484be701249cbe587c78da6edd556c2367e87a5c4230f06b0b4829dc75066d: Status 404 returned error can't find the container with id 84484be701249cbe587c78da6edd556c2367e87a5c4230f06b0b4829dc75066d Jul 10 19:33:56.206725 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:56.206690 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:56.206846 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:56.206770 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:56.206908 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:56.206840 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:33:56.798305 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:56.798270 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" event={"ID":"197355fb-227f-473d-abea-bbcec07103b6","Type":"ContainerStarted","Data":"f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da"} Jul 10 19:33:56.798305 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:56.798310 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" event={"ID":"197355fb-227f-473d-abea-bbcec07103b6","Type":"ContainerStarted","Data":"84484be701249cbe587c78da6edd556c2367e87a5c4230f06b0b4829dc75066d"} Jul 10 19:33:56.819488 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:56.819421 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" podStartSLOduration=1.819402262 podStartE2EDuration="1.819402262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:33:56.816898705 +0000 UTC m=+2013.567803596" watchObservedRunningTime="2026-07-10 19:33:56.819402262 +0000 UTC m=+2013.570307204" Jul 10 19:33:57.058389 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:57.058278 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:57.063639 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:57.063611 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:57.805007 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:57.804978 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:33:57.805158 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:33:57.805049 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:34:00.175674 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.175609 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:34:00.264727 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.264696 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-certs\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.264951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.264749 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-token\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.264951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.264770 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istiod-ca-cert\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.264951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.264813 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-socket\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.264951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.264835 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9m6b\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-kube-api-access-r9m6b\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.264951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.264877 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-data\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.264951 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.264950 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-credential-socket\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.265304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265002 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-podinfo\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.265304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265050 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-envoy\") pod \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\" (UID: \"1ade91d7-54dd-46bd-9f14-c0c2d44a665f\") " Jul 10 19:34:00.265304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265101 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:34:00.265304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265206 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:34:00.265304 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265297 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:34:00.265555 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265364 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:34:00.265555 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265461 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.265555 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265483 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.265555 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265499 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.265555 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.265512 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.266410 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.266379 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:34:00.267275 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.267253 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-kube-api-access-r9m6b" (OuterVolumeSpecName: "kube-api-access-r9m6b") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "kube-api-access-r9m6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:34:00.267523 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.267508 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:34:00.267710 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.267693 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:34:00.267752 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.267710 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ade91d7-54dd-46bd-9f14-c0c2d44a665f" (UID: "1ade91d7-54dd-46bd-9f14-c0c2d44a665f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:34:00.366356 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.366316 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.366356 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.366349 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.366356 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.366357 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.366356 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.366365 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9m6b\" (UniqueName: \"kubernetes.io/projected/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-kube-api-access-r9m6b\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.366637 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.366375 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ade91d7-54dd-46bd-9f14-c0c2d44a665f-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:34:00.819748 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.819711 2549 generic.go:356] "Generic (PLEG): container finished" podID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerID="8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63" exitCode=0 Jul 10 19:34:00.819940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.819754 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" event={"ID":"1ade91d7-54dd-46bd-9f14-c0c2d44a665f","Type":"ContainerDied","Data":"8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63"} Jul 10 19:34:00.819940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.819811 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" event={"ID":"1ade91d7-54dd-46bd-9f14-c0c2d44a665f","Type":"ContainerDied","Data":"8443de76be9fb666bf038d9483b7a75663c50720615f289f5c6d81d2b6ad2030"} Jul 10 19:34:00.819940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.819835 2549 scope.go:117] "RemoveContainer" containerID="8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63" Jul 10 19:34:00.819940 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.819755 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf" Jul 10 19:34:00.829879 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.829854 2549 scope.go:117] "RemoveContainer" containerID="8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd" Jul 10 19:34:00.838294 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.838276 2549 scope.go:117] "RemoveContainer" containerID="8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63" Jul 10 19:34:00.838558 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:34:00.838537 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63\": container with ID starting with 8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63 not found: ID does not exist" containerID="8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63" Jul 10 19:34:00.838613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.838565 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63"} err="failed to get container status \"8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63\": rpc error: code = NotFound desc = could not find container \"8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63\": container with ID starting with 8042d917fd91f68398ed6ea689f2dfb1684eab3232adf9c7ed0854a8969dee63 not found: ID does not exist" Jul 10 19:34:00.838613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.838582 2549 scope.go:117] "RemoveContainer" containerID="8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd" Jul 10 19:34:00.838916 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:34:00.838893 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd\": container with ID starting with 8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd not found: ID does not exist" containerID="8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd" Jul 10 19:34:00.838975 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.838922 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd"} err="failed to get container status \"8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd\": rpc error: code = NotFound desc = could not find container \"8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd\": container with ID starting with 8432415cb1fd9b14fb5486f01360ec4e93de24bb27fb9382ede2ab97c1bd5dcd not found: ID does not exist" Jul 10 19:34:00.850875 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.850850 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf"] Jul 10 19:34:00.858863 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:00.858838 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-094d6e-openshift-default-7d9888cbf7-xtnwf"] Jul 10 19:34:01.804827 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:01.804776 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" path="/var/lib/kubelet/pods/1ade91d7-54dd-46bd-9f14-c0c2d44a665f/volumes" Jul 10 19:34:03.792243 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792202 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf"] Jul 10 19:34:03.792709 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792691 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" Jul 10 19:34:03.792778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792712 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="49bb4952-6416-403d-ab19-75ae6c4e7209" containerName="istio-proxy" Jul 10 19:34:03.792778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792739 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" Jul 10 19:34:03.792778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792748 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" Jul 10 19:34:03.792778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792761 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" Jul 10 19:34:03.792778 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792767 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" Jul 10 19:34:03.792989 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792858 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" Jul 10 19:34:03.792989 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.792869 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ade91d7-54dd-46bd-9f14-c0c2d44a665f" containerName="istio-proxy" Jul 10 19:34:03.795851 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.795830 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.798174 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.798146 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b02417-openshift-default-dockercfg-lnlqk\"" Jul 10 19:34:03.806764 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.806740 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf"] Jul 10 19:34:03.899991 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.899949 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-socket\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.900158 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.900014 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-credential-socket\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.900222 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.900205 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-token\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.900303 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.900287 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-certs\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.900368 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.900318 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-podinfo\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.900482 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.900455 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfpq\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-kube-api-access-vgfpq\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.900552 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.900502 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-data\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.900607 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.900576 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-envoy\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:03.900663 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:03.900623 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istiod-ca-cert\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001563 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001523 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-envoy\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001563 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001570 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istiod-ca-cert\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001619 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-socket\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001649 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-credential-socket\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001675 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-token\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001730 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-certs\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001756 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-podinfo\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001833 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfpq\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-kube-api-access-vgfpq\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.001886 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.001859 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-data\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.002246 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.002154 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-socket\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.002246 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.002199 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-credential-socket\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.002246 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.002223 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-certs\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.002561 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.002358 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-data\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.002561 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.002477 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istiod-ca-cert\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.004090 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.004067 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-envoy\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.004339 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.004321 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-podinfo\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.016577 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.016541 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-token\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.016671 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.016623 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfpq\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-kube-api-access-vgfpq\") pod \"e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.108043 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.108006 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:04.248887 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.248861 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf"] Jul 10 19:34:04.251292 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:34:04.251259 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb9e83a_061d_43a8_842b_b3d8c05aea6d.slice/crio-20c601b60b20a5f2b3b6b27207102cdad4290cd877daefdcab246cb498472b54 WatchSource:0}: Error finding container 20c601b60b20a5f2b3b6b27207102cdad4290cd877daefdcab246cb498472b54: Status 404 returned error can't find the container with id 20c601b60b20a5f2b3b6b27207102cdad4290cd877daefdcab246cb498472b54 Jul 10 19:34:04.253438 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.253408 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:04.253496 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.253470 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:04.253539 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.253500 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:04.844993 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.844950 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" event={"ID":"3eb9e83a-061d-43a8-842b-b3d8c05aea6d","Type":"ContainerStarted","Data":"74ed16ba38001915a0b76a5d5e052e1e67cd10a248303b6c6ba0b319b34f97d1"} Jul 10 19:34:04.845409 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.845022 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" event={"ID":"3eb9e83a-061d-43a8-842b-b3d8c05aea6d","Type":"ContainerStarted","Data":"20c601b60b20a5f2b3b6b27207102cdad4290cd877daefdcab246cb498472b54"} Jul 10 19:34:04.867986 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:04.867938 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" podStartSLOduration=1.867923201 podStartE2EDuration="1.867923201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:04.866406016 +0000 UTC m=+2021.617310894" watchObservedRunningTime="2026-07-10 19:34:04.867923201 +0000 UTC m=+2021.618828074" Jul 10 19:34:05.108701 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:05.108621 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:05.113382 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:05.113357 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:05.854386 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:05.854209 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:05.855325 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:05.855301 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:12.805874 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:12.805831 2549 patch_prober.go:28] interesting pod/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 19:34:12.806368 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:12.805894 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 19:34:12.897856 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:12.897817 2549 generic.go:356] "Generic (PLEG): container finished" podID="197355fb-227f-473d-abea-bbcec07103b6" containerID="f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da" exitCode=0 Jul 10 19:34:12.898034 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:12.897886 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" event={"ID":"197355fb-227f-473d-abea-bbcec07103b6","Type":"ContainerDied","Data":"f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da"} Jul 10 19:34:12.898335 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:12.898319 2549 scope.go:117] "RemoveContainer" containerID="f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da" Jul 10 19:34:12.899538 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:12.899509 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:12.899608 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:12.899593 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:12.899657 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:12.899645 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:13.904936 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:13.904899 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" event={"ID":"197355fb-227f-473d-abea-bbcec07103b6","Type":"ContainerStarted","Data":"a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e"} Jul 10 19:34:14.058434 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:14.058039 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:34:14.058708 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:14.058565 2549 patch_prober.go:28] interesting pod/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 19:34:14.058708 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:14.058673 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 19:34:15.058001 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:15.057956 2549 patch_prober.go:28] interesting pod/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 19:34:15.058508 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:15.058024 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 19:34:16.058306 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:16.058265 2549 patch_prober.go:28] interesting pod/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 19:34:16.058948 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:16.058912 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 19:34:17.058333 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:17.058256 2549 patch_prober.go:28] interesting pod/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 10 19:34:17.058764 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:17.058479 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 10 19:34:18.061981 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:18.061949 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:34:18.062396 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:18.062305 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:34:18.063305 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:18.063286 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:34:19.937458 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:19.937418 2549 generic.go:356] "Generic (PLEG): container finished" podID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerID="74ed16ba38001915a0b76a5d5e052e1e67cd10a248303b6c6ba0b319b34f97d1" exitCode=0 Jul 10 19:34:19.937971 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:19.937496 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" event={"ID":"3eb9e83a-061d-43a8-842b-b3d8c05aea6d","Type":"ContainerDied","Data":"74ed16ba38001915a0b76a5d5e052e1e67cd10a248303b6c6ba0b319b34f97d1"} Jul 10 19:34:19.938043 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:19.938014 2549 scope.go:117] "RemoveContainer" containerID="74ed16ba38001915a0b76a5d5e052e1e67cd10a248303b6c6ba0b319b34f97d1" Jul 10 19:34:19.939072 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:19.939043 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:19.939178 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:19.939150 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:19.939244 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:19.939218 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 19:34:20.854138 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:20.854088 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:20.947245 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:20.947194 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" event={"ID":"3eb9e83a-061d-43a8-842b-b3d8c05aea6d","Type":"ContainerStarted","Data":"d39d319c24cf9c41f093f9acd12272958ee596b5f2f4b557795429cd83f9739a"} Jul 10 19:34:21.108545 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:21.108455 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:21.109909 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:21.109881 2549 patch_prober.go:28] interesting pod/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 10 19:34:21.110043 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:21.109958 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 10 19:34:22.108442 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:22.108406 2549 patch_prober.go:28] interesting pod/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 10 19:34:22.108865 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:22.108461 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 10 19:34:23.109193 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:23.109154 2549 patch_prober.go:28] interesting pod/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 10 19:34:23.109574 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:23.109212 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 10 19:34:24.108624 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:24.108584 2549 patch_prober.go:28] interesting pod/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 10 19:34:24.108833 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:24.108647 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 10 19:34:25.113111 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:25.113071 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:25.113554 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:25.113380 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:34:25.114184 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:34:25.114163 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:35:23.867466 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:35:23.867428 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:35:23.873010 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:35:23.872987 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:35:23.877129 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:35:23.877110 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:35:23.881683 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:35:23.881664 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:40:17.955208 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:17.955177 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk"] Jul 10 19:40:17.955837 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:17.955413 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" containerID="cri-o://a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e" gracePeriod=30 Jul 10 19:40:18.064461 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:18.064426 2549 patch_prober.go:28] interesting pod/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 19:40:18.064647 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:18.064499 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:40:23.205457 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.205427 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:40:23.307068 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.306988 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197355fb-227f-473d-abea-bbcec07103b6-istio-podinfo\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307068 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307041 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197355fb-227f-473d-abea-bbcec07103b6-istiod-ca-cert\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307068 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307062 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-socket\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307336 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307081 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6jts\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-kube-api-access-z6jts\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307336 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307150 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-credential-socket\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307336 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307175 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-envoy\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307336 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307204 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-istio-token\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307336 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307248 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-certs\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307336 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307278 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-data\") pod \"197355fb-227f-473d-abea-bbcec07103b6\" (UID: \"197355fb-227f-473d-abea-bbcec07103b6\") " Jul 10 19:40:23.307696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307356 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:23.307696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307365 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197355fb-227f-473d-abea-bbcec07103b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:40:23.307696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307479 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:23.307696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307651 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.307696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307671 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/197355fb-227f-473d-abea-bbcec07103b6-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.307696 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307684 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.308196 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.307960 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:23.308548 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.308522 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:23.309487 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.309455 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/197355fb-227f-473d-abea-bbcec07103b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:40:23.309594 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.309524 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-kube-api-access-z6jts" (OuterVolumeSpecName: "kube-api-access-z6jts") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "kube-api-access-z6jts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:23.309654 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.309610 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:23.309892 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.309878 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "197355fb-227f-473d-abea-bbcec07103b6" (UID: "197355fb-227f-473d-abea-bbcec07103b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:23.408592 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.408561 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.408592 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.408584 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.408592 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.408593 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.408592 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.408600 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/197355fb-227f-473d-abea-bbcec07103b6-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.408867 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.408608 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/197355fb-227f-473d-abea-bbcec07103b6-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.408867 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.408616 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6jts\" (UniqueName: \"kubernetes.io/projected/197355fb-227f-473d-abea-bbcec07103b6-kube-api-access-z6jts\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:23.739808 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.739761 2549 generic.go:356] "Generic (PLEG): container finished" podID="197355fb-227f-473d-abea-bbcec07103b6" containerID="a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e" exitCode=0 Jul 10 19:40:23.739970 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.739816 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" Jul 10 19:40:23.739970 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.739825 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" event={"ID":"197355fb-227f-473d-abea-bbcec07103b6","Type":"ContainerDied","Data":"a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e"} Jul 10 19:40:23.739970 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.739901 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk" event={"ID":"197355fb-227f-473d-abea-bbcec07103b6","Type":"ContainerDied","Data":"84484be701249cbe587c78da6edd556c2367e87a5c4230f06b0b4829dc75066d"} Jul 10 19:40:23.739970 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.739922 2549 scope.go:117] "RemoveContainer" containerID="a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e" Jul 10 19:40:23.749242 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.749225 2549 scope.go:117] "RemoveContainer" containerID="f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da" Jul 10 19:40:23.757072 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.757057 2549 scope.go:117] "RemoveContainer" containerID="a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e" Jul 10 19:40:23.757315 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:40:23.757291 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e\": container with ID starting with a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e not found: ID does not exist" containerID="a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e" Jul 10 19:40:23.757358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.757324 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e"} err="failed to get container status \"a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e\": rpc error: code = NotFound desc = could not find container \"a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e\": container with ID starting with a6c2402b962d02c3f874d151e6e48d252a2b1250dd94d82133530e730274683e not found: ID does not exist" Jul 10 19:40:23.757358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.757339 2549 scope.go:117] "RemoveContainer" containerID="f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da" Jul 10 19:40:23.757569 ip-10-0-140-210 kubenswrapper[2549]: E0710 19:40:23.757554 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da\": container with ID starting with f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da not found: ID does not exist" containerID="f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da" Jul 10 19:40:23.757615 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.757581 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da"} err="failed to get container status \"f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da\": rpc error: code = NotFound desc = could not find container \"f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da\": container with ID starting with f41c40c946b33c2e83562972435b5f4740644e2d71826dec4dd03f2c9dc1f4da not found: ID does not exist" Jul 10 19:40:23.764176 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.764153 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk"] Jul 10 19:40:23.770629 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.770609 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b1f450-openshift-default-5c9d66cff-c2lgk"] Jul 10 19:40:23.804066 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.804042 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="197355fb-227f-473d-abea-bbcec07103b6" path="/var/lib/kubelet/pods/197355fb-227f-473d-abea-bbcec07103b6/volumes" Jul 10 19:40:23.895086 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.895063 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:40:23.899786 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.899771 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:40:23.911974 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.911952 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:40:23.916599 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:23.916581 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cz8np_c0b5f626-0e73-46e6-8a00-b127ccf339ee/ovn-acl-logging/0.log" Jul 10 19:40:25.591316 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:25.591285 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf"] Jul 10 19:40:25.591686 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:25.591533 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerName="istio-proxy" containerID="cri-o://d39d319c24cf9c41f093f9acd12272958ee596b5f2f4b557795429cd83f9739a" gracePeriod=30 Jul 10 19:40:30.778197 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.778168 2549 generic.go:356] "Generic (PLEG): container finished" podID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerID="d39d319c24cf9c41f093f9acd12272958ee596b5f2f4b557795429cd83f9739a" exitCode=0 Jul 10 19:40:30.778522 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.778254 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" event={"ID":"3eb9e83a-061d-43a8-842b-b3d8c05aea6d","Type":"ContainerDied","Data":"d39d319c24cf9c41f093f9acd12272958ee596b5f2f4b557795429cd83f9739a"} Jul 10 19:40:30.778522 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.778366 2549 scope.go:117] "RemoveContainer" containerID="74ed16ba38001915a0b76a5d5e052e1e67cd10a248303b6c6ba0b319b34f97d1" Jul 10 19:40:30.840195 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.840176 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:40:30.875427 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.875377 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-socket\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.875667 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.875640 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:30.975930 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.975908 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-data\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.976041 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.975935 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-token\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.976041 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.975965 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-certs\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.976041 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.975985 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-podinfo\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.976041 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976025 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-envoy\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.976264 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976057 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgfpq\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-kube-api-access-vgfpq\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.976264 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976079 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istiod-ca-cert\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.976264 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976106 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-credential-socket\") pod \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\" (UID: \"3eb9e83a-061d-43a8-842b-b3d8c05aea6d\") " Jul 10 19:40:30.976264 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976233 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:30.976450 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976351 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-certs\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:30.976450 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976369 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-workload-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:30.976450 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976404 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:30.976450 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976430 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:40:30.976949 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.976872 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:30.978148 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.978127 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:30.978233 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.978164 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 19:40:30.978358 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.978344 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:30.978661 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:30.978633 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-kube-api-access-vgfpq" (OuterVolumeSpecName: "kube-api-access-vgfpq") pod "3eb9e83a-061d-43a8-842b-b3d8c05aea6d" (UID: "3eb9e83a-061d-43a8-842b-b3d8c05aea6d"). InnerVolumeSpecName "kube-api-access-vgfpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:31.076672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.076644 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-data\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:31.076672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.076665 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-token\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:31.076672 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.076674 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-podinfo\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:31.076926 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.076681 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istio-envoy\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:31.076926 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.076689 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgfpq\" (UniqueName: \"kubernetes.io/projected/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-kube-api-access-vgfpq\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:31.076926 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.076697 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-istiod-ca-cert\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:31.076926 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.076704 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb9e83a-061d-43a8-842b-b3d8c05aea6d-credential-socket\") on node \"ip-10-0-140-210.ec2.internal\" DevicePath \"\"" Jul 10 19:40:31.355343 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.355320 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-98f7fc48c-ll6c7_3eb2b148-4536-4842-8a49-1a67774f9d04/manager/0.log" Jul 10 19:40:31.486411 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.486390 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bcngx_405d5e66-7968-48e0-a384-b7cd60fd4db6/server/0.log" Jul 10 19:40:31.594723 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.594698 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kj9l7_49146843-8c17-438b-b5fd-231d5c18f7f3/manager/0.log" Jul 10 19:40:31.785560 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.785489 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" event={"ID":"3eb9e83a-061d-43a8-842b-b3d8c05aea6d","Type":"ContainerDied","Data":"20c601b60b20a5f2b3b6b27207102cdad4290cd877daefdcab246cb498472b54"} Jul 10 19:40:31.785560 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.785511 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf" Jul 10 19:40:31.785560 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.785538 2549 scope.go:117] "RemoveContainer" containerID="d39d319c24cf9c41f093f9acd12272958ee596b5f2f4b557795429cd83f9739a" Jul 10 19:40:31.823164 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.823138 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf"] Jul 10 19:40:31.832953 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:31.832930 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b02417-openshift-default-67786649bc-c2tnf"] Jul 10 19:40:32.005628 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:32.005607 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7668cd484b-hzpr9_f57f5f88-b93a-4886-8ad8-454598cbea0b/copy-manifests/0.log" Jul 10 19:40:32.012276 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:32.012254 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7668cd484b-hzpr9_f57f5f88-b93a-4886-8ad8-454598cbea0b/manager/0.log" Jul 10 19:40:32.345441 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:32.345416 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6695c44845-thjc5_07411a59-ce66-4534-8a33-80da0ea8e051/maas-api/0.log" Jul 10 19:40:33.806610 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:33.806566 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" path="/var/lib/kubelet/pods/3eb9e83a-061d-43a8-842b-b3d8c05aea6d/volumes" Jul 10 19:40:33.965875 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:33.965845 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-v5nn2_4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2/manager/0.log" Jul 10 19:40:34.343608 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:34.343575 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-c6qh9_7f189776-e68f-4fc6-90b5-e69c4914ea55/manager/0.log" Jul 10 19:40:34.580335 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:34.580306 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6mhsf_79cbea14-e7aa-448a-9e87-cb4f9005dc31/manager/0.log" Jul 10 19:40:34.922729 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:34.922705 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678rpldg_f4238c24-1222-4d9c-bc57-23759dbaf816/istio-proxy/0.log" Jul 10 19:40:35.039135 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:35.039112 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nfgjl_267d0dc5-ed6e-4977-889e-9c620744d07d/discovery/0.log" Jul 10 19:40:35.267089 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:35.267008 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b4fc6b9d-c2ngv_54995f91-4f93-4673-a906-091f7cd21698/kube-auth-proxy/0.log" Jul 10 19:40:35.504039 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:35.504012 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ph9cl_183ef325-6e3f-4700-8913-1de86302ea0a/payload-pre-processing/0.log" Jul 10 19:40:35.616613 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:35.616584 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xsgpq_5b1ea410-9c31-4085-ad65-c38124158b9e/payload-processing/0.log" Jul 10 19:40:36.268517 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:36.268492 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-2hcvn_580fe534-ceef-40f1-9d9a-a46fe84457b9/main/0.log" Jul 10 19:40:36.514171 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:36.514143 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4tgcs_93dd9324-860a-40f3-a341-57572b6fcffa/main/0.log" Jul 10 19:40:43.269397 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:43.269368 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qhkzc_2f888fa4-b9d3-45dc-bc95-9801d7ecef40/global-pull-secret-syncer/0.log" Jul 10 19:40:43.374478 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:43.374453 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mddxb_b0be705c-d948-4f4e-aa76-5baab302ac9c/konnectivity-agent/0.log" Jul 10 19:40:43.498656 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:43.498631 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-210.ec2.internal_a2ad6244da66dd8524b1bacda2ae8771/haproxy/0.log" Jul 10 19:40:49.200847 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:49.200779 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-v5nn2_4bb2f3f0-2fb1-4c8c-b6c4-31e1e26eccb2/manager/0.log" Jul 10 19:40:49.401154 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:49.401122 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-c6qh9_7f189776-e68f-4fc6-90b5-e69c4914ea55/manager/0.log" Jul 10 19:40:49.503074 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:49.503001 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6mhsf_79cbea14-e7aa-448a-9e87-cb4f9005dc31/manager/0.log" Jul 10 19:40:51.525896 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:51.525871 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-wzlwt_2db73e7c-4a11-4d2b-ac12-65ec881419f8/monitoring-plugin/0.log" Jul 10 19:40:51.640348 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:51.640320 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqlrc_cd66431b-b049-486b-b194-31e528e42fda/node-exporter/0.log" Jul 10 19:40:51.663644 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:51.663626 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqlrc_cd66431b-b049-486b-b194-31e528e42fda/kube-rbac-proxy/0.log" Jul 10 19:40:51.687650 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:51.687632 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dqlrc_cd66431b-b049-486b-b194-31e528e42fda/init-textfile/0.log" Jul 10 19:40:53.429943 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.429915 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7"] Jul 10 19:40:53.430272 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.430255 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerName="istio-proxy" Jul 10 19:40:53.430272 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.430265 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerName="istio-proxy" Jul 10 19:40:53.430341 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.430274 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" Jul 10 19:40:53.430341 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.430280 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" Jul 10 19:40:53.430405 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.430340 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb9e83a-061d-43a8-842b-b3d8c05aea6d" containerName="istio-proxy" Jul 10 19:40:53.430405 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.430349 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="197355fb-227f-473d-abea-bbcec07103b6" containerName="istio-proxy" Jul 10 19:40:53.434999 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.434980 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.437474 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.437458 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z44r6\"/\"default-dockercfg-q7qdh\"" Jul 10 19:40:53.437576 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.437476 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z44r6\"/\"kube-root-ca.crt\"" Jul 10 19:40:53.438312 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.438293 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z44r6\"/\"openshift-service-ca.crt\"" Jul 10 19:40:53.444215 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.444195 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7"] Jul 10 19:40:53.548942 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.548918 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-lib-modules\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.549059 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.548980 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-sys\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.549059 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.549049 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-podres\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.549153 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.549078 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-proc\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.549153 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.549099 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4g8\" (UniqueName: \"kubernetes.io/projected/9a9da299-dc0a-4b70-83fa-9599f1472e63-kube-api-access-bx4g8\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650219 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650195 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-lib-modules\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650317 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650250 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-sys\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650317 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650280 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-podres\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650317 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650297 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-proc\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650317 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650312 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4g8\" (UniqueName: \"kubernetes.io/projected/9a9da299-dc0a-4b70-83fa-9599f1472e63-kube-api-access-bx4g8\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650559 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650362 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-lib-modules\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650559 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650387 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-proc\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650559 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650386 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-sys\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.650559 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.650529 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9a9da299-dc0a-4b70-83fa-9599f1472e63-podres\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.667716 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.667691 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4g8\" (UniqueName: \"kubernetes.io/projected/9a9da299-dc0a-4b70-83fa-9599f1472e63-kube-api-access-bx4g8\") pod \"perf-node-gather-daemonset-klxh7\" (UID: \"9a9da299-dc0a-4b70-83fa-9599f1472e63\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.731911 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.731862 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-99qsw_e85f93bd-0c5e-403f-b443-9cc24790e26c/networking-console-plugin/0.log" Jul 10 19:40:53.745043 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.745026 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:53.881952 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:53.881929 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7"] Jul 10 19:40:53.883888 ip-10-0-140-210 kubenswrapper[2549]: W0710 19:40:53.883861 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9a9da299_dc0a_4b70_83fa_9599f1472e63.slice/crio-4ce1d98115c057c6f9fbd85ade2acd327c30593999e107d0b898f4ec2e150783 WatchSource:0}: Error finding container 4ce1d98115c057c6f9fbd85ade2acd327c30593999e107d0b898f4ec2e150783: Status 404 returned error can't find the container with id 4ce1d98115c057c6f9fbd85ade2acd327c30593999e107d0b898f4ec2e150783 Jul 10 19:40:54.305608 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:54.305578 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/1.log" Jul 10 19:40:54.310982 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:54.310943 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-kpccd_b7d9cb01-a594-425b-ab20-68088641139a/console-operator/2.log" Jul 10 19:40:54.814066 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:54.814040 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c655fb948-glxvt_23f77890-1285-45ac-9195-a3502235a23d/console/0.log" Jul 10 19:40:54.888528 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:54.888497 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" event={"ID":"9a9da299-dc0a-4b70-83fa-9599f1472e63","Type":"ContainerStarted","Data":"f9c332151bfda750fa53eb14954ea3012416270fb67c4dcef3a5157a82f2c531"} Jul 10 19:40:54.888703 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:54.888551 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" event={"ID":"9a9da299-dc0a-4b70-83fa-9599f1472e63","Type":"ContainerStarted","Data":"4ce1d98115c057c6f9fbd85ade2acd327c30593999e107d0b898f4ec2e150783"} Jul 10 19:40:54.888703 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:54.888598 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:40:54.907432 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:54.907384 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" podStartSLOduration=1.907371438 podStartE2EDuration="1.907371438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:40:54.904940997 +0000 UTC m=+2431.655845910" watchObservedRunningTime="2026-07-10 19:40:54.907371438 +0000 UTC m=+2431.658276310" Jul 10 19:40:55.452949 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:55.452918 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-p59jv_92a9c9d5-5f40-445d-9337-c8216a134c1d/volume-data-source-validator/0.log" Jul 10 19:40:56.761596 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:56.761570 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ksmpt_a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186/dns/0.log" Jul 10 19:40:56.787293 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:56.787269 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ksmpt_a4ca9b75-92b4-4cf5-b2c9-218d1c4f2186/kube-rbac-proxy/0.log" Jul 10 19:40:56.816107 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:56.816084 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gclsw_93b1de2e-aef6-4424-808f-dbecaeff1c47/dns-node-resolver/0.log" Jul 10 19:40:57.485184 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:57.485139 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c87d7859f-vvbzf_83f8db00-c888-4ce8-8637-bb43dc72920e/registry/0.log" Jul 10 19:40:57.509250 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:57.509230 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4zngt_fce47570-05bd-436a-91d8-9799b73e968d/node-ca/0.log" Jul 10 19:40:58.589371 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:58.589346 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678rpldg_f4238c24-1222-4d9c-bc57-23759dbaf816/istio-proxy/0.log" Jul 10 19:40:58.701199 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:58.701171 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nfgjl_267d0dc5-ed6e-4977-889e-9c620744d07d/discovery/0.log" Jul 10 19:40:58.751712 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:58.751688 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b4fc6b9d-c2ngv_54995f91-4f93-4673-a906-091f7cd21698/kube-auth-proxy/0.log" Jul 10 19:40:58.874453 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:58.874385 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ph9cl_183ef325-6e3f-4700-8913-1de86302ea0a/payload-pre-processing/0.log" Jul 10 19:40:58.904573 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:58.904548 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xsgpq_5b1ea410-9c31-4085-ad65-c38124158b9e/payload-processing/0.log" Jul 10 19:40:59.693199 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:40:59.693171 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5bcjs_deff8a3a-41de-4b18-8f3e-a29843bab383/serve-healthcheck-canary/0.log" Jul 10 19:41:00.243073 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:00.243049 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8b6wq_ae7dfa1d-acb5-4192-91f8-241e98c91f61/kube-rbac-proxy/0.log" Jul 10 19:41:00.266651 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:00.266631 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8b6wq_ae7dfa1d-acb5-4192-91f8-241e98c91f61/exporter/0.log" Jul 10 19:41:00.291537 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:00.291519 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8b6wq_ae7dfa1d-acb5-4192-91f8-241e98c91f61/extractor/0.log" Jul 10 19:41:00.902739 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:00.902715 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-klxh7" Jul 10 19:41:03.263128 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:03.263098 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-98f7fc48c-ll6c7_3eb2b148-4536-4842-8a49-1a67774f9d04/manager/0.log" Jul 10 19:41:03.291361 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:03.291337 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bcngx_405d5e66-7968-48e0-a384-b7cd60fd4db6/server/0.log" Jul 10 19:41:03.360322 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:03.360303 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kj9l7_49146843-8c17-438b-b5fd-231d5c18f7f3/manager/0.log" Jul 10 19:41:03.597734 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:03.597637 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7668cd484b-hzpr9_f57f5f88-b93a-4886-8ad8-454598cbea0b/manager/0.log" Jul 10 19:41:03.621078 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:03.621053 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7668cd484b-hzpr9_f57f5f88-b93a-4886-8ad8-454598cbea0b/copy-manifests/0.log" Jul 10 19:41:04.057813 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:04.057713 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678rpldg_f4238c24-1222-4d9c-bc57-23759dbaf816/istio-proxy/0.log" Jul 10 19:41:04.156373 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:04.156343 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nfgjl_267d0dc5-ed6e-4977-889e-9c620744d07d/discovery/0.log" Jul 10 19:41:04.184625 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:04.184600 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b4fc6b9d-c2ngv_54995f91-4f93-4673-a906-091f7cd21698/kube-auth-proxy/0.log" Jul 10 19:41:04.282637 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:04.282613 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ph9cl_183ef325-6e3f-4700-8913-1de86302ea0a/payload-pre-processing/0.log" Jul 10 19:41:04.305650 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:04.305628 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xsgpq_5b1ea410-9c31-4085-ad65-c38124158b9e/payload-processing/0.log" Jul 10 19:41:05.608445 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:05.608416 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678rpldg_f4238c24-1222-4d9c-bc57-23759dbaf816/istio-proxy/0.log" Jul 10 19:41:05.713161 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:05.713128 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-nfgjl_267d0dc5-ed6e-4977-889e-9c620744d07d/discovery/0.log" Jul 10 19:41:05.742556 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:05.742534 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b4fc6b9d-c2ngv_54995f91-4f93-4673-a906-091f7cd21698/kube-auth-proxy/0.log" Jul 10 19:41:05.852576 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:05.852552 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ph9cl_183ef325-6e3f-4700-8913-1de86302ea0a/payload-pre-processing/0.log" Jul 10 19:41:05.874075 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:05.874022 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xsgpq_5b1ea410-9c31-4085-ad65-c38124158b9e/payload-processing/0.log" Jul 10 19:41:10.338157 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:10.338126 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-7g2d7_2c63a7aa-df11-4c44-bdc5-e19b075bf311/kube-storage-version-migrator-operator/1.log" Jul 10 19:41:10.339112 ip-10-0-140-210 kubenswrapper[2549]: I0710 19:41:10.339096 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-7g2d7_2c63a7aa-df11-4c44-bdc5-e19b075bf311/kube-storage-version-migrator-operator/0.log"