Jul 17 09:26:56.230049 ip-10-0-133-236 systemd[1]: Starting Kubernetes Kubelet... Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.649817 2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653666 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653678 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653684 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653688 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653692 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653695 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653698 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 09:26:56.754138 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653701 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 09:26:56.722293 ip-10-0-133-236 systemd[1]: Started Kubernetes Kubelet. Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653704 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653707 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653709 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653726 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653729 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653737 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653740 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653743 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653746 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653749 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653751 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653754 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653757 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653759 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653762 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653764 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653767 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653770 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 09:26:56.755191 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653774 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653778 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653781 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653784 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653787 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653790 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653793 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653796 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653798 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653801 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653804 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653806 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653809 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653812 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653814 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653817 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653819 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653823 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653825 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653828 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 09:26:56.756887 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653831 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653834 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653836 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653839 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653841 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653844 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653847 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653849 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653852 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653854 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653858 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653861 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653863 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653866 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653870 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653873 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653876 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653879 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653882 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653885 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 09:26:56.757552 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653888 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653890 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653893 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653895 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653898 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653902 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653905 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653907 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653910 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653913 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653915 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653919 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653923 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653926 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653928 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653931 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653933 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653936 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653938 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 09:26:56.758182 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653941 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653943 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653946 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653948 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653951 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653954 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653956 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653959 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653962 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653965 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653967 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653970 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653973 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.653976 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654695 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654703 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654707 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654710 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654728 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654732 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 09:26:58.019807 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654735 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654738 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654740 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654743 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654746 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654749 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654752 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654755 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654757 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654760 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654762 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654765 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654768 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654771 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654775 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654779 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654782 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654786 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654790 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 09:26:58.020946 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654793 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654796 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654799 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654802 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654805 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654807 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654810 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654813 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654815 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654818 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654821 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654824 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654826 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654830 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654832 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654835 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654837 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654840 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654843 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654845 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654848 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 09:26:58.021781 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654851 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654853 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654856 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654858 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654861 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654864 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654867 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654870 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654872 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654875 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654878 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654880 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654883 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654886 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654890 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654892 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654895 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654898 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654901 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654904 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 09:26:58.022824 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654907 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654909 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654912 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654916 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654919 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654923 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654926 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654929 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654932 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654934 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654937 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654940 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654944 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654946 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654949 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654952 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654955 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654957 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654960 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 09:26:58.023844 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654962 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654965 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654968 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654970 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654975 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654978 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654981 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654983 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654986 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654989 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654991 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654994 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654997 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.654999 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655393 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655398 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655401 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655404 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655408 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 09:26:58.024455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655411 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655413 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655416 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655419 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655421 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655424 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655427 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655429 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655431 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655434 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655437 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655439 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655442 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655444 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655447 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655450 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655452 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655455 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655457 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655460 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 09:26:58.025065 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655462 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655465 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655468 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655470 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655472 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655475 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655477 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655483 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655486 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655490 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655493 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655495 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655499 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655501 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655504 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655507 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655510 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655514 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655517 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655519 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 09:26:58.025905 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655522 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655524 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655527 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655529 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655532 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655536 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655540 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655543 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655546 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655549 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655552 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655555 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655558 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655560 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655565 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655569 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655572 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655575 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655577 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 09:26:58.026912 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655583 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655586 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655590 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655593 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655595 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655598 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655601 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655603 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655607 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655610 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655612 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655615 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655617 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655620 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655622 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655625 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655627 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655630 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655633 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655636 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 09:26:58.027533 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655638 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655640 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655643 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655645 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655648 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655650 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655653 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655656 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655658 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655661 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655663 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655666 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655670 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655673 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.655676 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655762 2584 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655769 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655777 2584 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655781 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655786 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655789 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 09:26:58.028147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655794 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655798 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655801 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655805 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655808 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655811 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655814 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655817 2584 flags.go:64] FLAG: --cgroup-root="" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655820 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655823 2584 flags.go:64] FLAG: --client-ca-file="" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655826 2584 flags.go:64] FLAG: --cloud-provider="external" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655829 2584 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655890 2584 flags.go:64] FLAG: --cluster-domain="" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655893 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655897 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655900 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655903 2584 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655907 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655910 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655913 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655917 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655920 2584 flags.go:64] FLAG: --contention-profiling="false" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655923 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655927 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 09:26:58.028722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655931 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655935 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655939 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655942 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655946 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655948 2584 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655951 2584 flags.go:64] FLAG: --enable-server="true" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655955 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655960 2584 flags.go:64] FLAG: --event-burst="100" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655963 2584 flags.go:64] FLAG: --event-qps="50" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655966 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655969 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655972 2584 flags.go:64] FLAG: --eviction-hard="" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655976 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655979 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655982 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655985 2584 flags.go:64] FLAG: --eviction-soft="" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655988 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655991 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655994 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.655997 2584 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656000 2584 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656003 2584 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656006 2584 flags.go:64] FLAG: --feature-gates="" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656011 2584 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 09:26:58.029543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656014 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656016 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656019 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656022 2584 flags.go:64] FLAG: --healthz-port="10248" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656025 2584 flags.go:64] FLAG: --help="false" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656028 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656035 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656038 2584 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656043 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656046 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656050 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656053 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656056 2584 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656059 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656062 2584 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656066 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656069 2584 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656071 2584 flags.go:64] FLAG: --kube-reserved="" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656074 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656077 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656080 2584 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656083 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656086 2584 flags.go:64] FLAG: --lock-file="" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656089 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 09:26:58.030226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656092 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656095 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656100 2584 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656103 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656106 2584 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656109 2584 flags.go:64] FLAG: --logging-format="text" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656112 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656115 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656118 2584 flags.go:64] FLAG: --manifest-url="" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656121 2584 flags.go:64] FLAG: --manifest-url-header="" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656125 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656128 2584 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656132 2584 flags.go:64] FLAG: --max-pods="110" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656135 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656140 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656143 2584 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656147 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656150 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656153 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656156 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656164 2584 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656167 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656170 2584 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 09:26:58.030946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656173 2584 flags.go:64] FLAG: --pod-cidr="" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656177 2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656182 2584 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656185 2584 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656189 2584 flags.go:64] FLAG: --pods-per-core="0" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656192 2584 flags.go:64] FLAG: --port="10250" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656195 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656198 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05550cddd50b2737d" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656201 2584 flags.go:64] FLAG: --qos-reserved="" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656204 2584 flags.go:64] FLAG: --read-only-port="10255" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656207 2584 flags.go:64] FLAG: --register-node="true" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656210 2584 flags.go:64] FLAG: --register-with-taints="" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656214 2584 flags.go:64] FLAG: --registry-burst="10" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656216 2584 flags.go:64] FLAG: --registry-qps="5" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656219 2584 flags.go:64] FLAG: --reserved-cpus="" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656222 2584 flags.go:64] FLAG: --reserved-memory="" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656225 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656229 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656232 2584 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656235 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656238 2584 flags.go:64] FLAG: --runonce="false" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656240 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656244 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656247 2584 flags.go:64] FLAG: --seccomp-default="false" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656251 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 09:26:58.031656 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656255 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656259 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656262 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656265 2584 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656268 2584 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656272 2584 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656275 2584 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656277 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656281 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656284 2584 flags.go:64] FLAG: --system-cgroups="" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656287 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656293 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656295 2584 flags.go:64] FLAG: --tls-cert-file="" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656299 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656303 2584 flags.go:64] FLAG: --tls-min-version="" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656306 2584 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656308 2584 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656311 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656314 2584 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656317 2584 flags.go:64] FLAG: --v="2" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656321 2584 flags.go:64] FLAG: --version="false" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656325 2584 flags.go:64] FLAG: --vmodule="" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656329 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.656332 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656465 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 09:26:58.032351 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656469 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656472 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656475 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656478 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656481 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656485 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656490 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656494 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656497 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656509 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656512 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656515 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656517 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656520 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656523 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656526 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656528 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656532 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656535 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656538 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 09:26:58.033339 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656540 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656543 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656545 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656548 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656550 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656553 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656556 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656558 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656560 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656563 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656565 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656568 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656571 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656573 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656576 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656578 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656581 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656584 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656588 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656592 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 09:26:58.033947 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656594 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656598 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656602 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656605 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656607 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656610 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656613 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656616 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656618 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656621 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656624 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656626 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656629 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656632 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656635 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656637 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656640 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656642 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656645 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656647 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 09:26:58.232915 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656650 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656653 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656656 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656658 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656662 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656665 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656668 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656671 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656674 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656677 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656682 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656687 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656690 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656693 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656696 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656699 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656702 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656705 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656708 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 09:26:58.233696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656710 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656739 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656742 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656744 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656748 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656750 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656753 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656756 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656759 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656761 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656764 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656766 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656769 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656772 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656774 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656777 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656781 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656785 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 09:26:58.234263 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.656788 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.657291 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.664299 2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.664316 2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.664334 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.664340 2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664395 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664400 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664404 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664407 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664410 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664413 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664416 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664419 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 09:26:58.234791 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664421 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664423 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664426 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664429 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664432 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664434 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664437 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664440 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664442 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664445 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664448 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664450 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664454 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664456 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664459 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664462 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664465 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664467 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664470 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664476 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 09:26:58.235201 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664479 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664482 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664485 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664488 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664490 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664493 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664495 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664498 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664510 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664513 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664515 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664518 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664521 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664523 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664526 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664528 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664531 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664534 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664536 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664539 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 09:26:58.235792 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664541 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664546 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664549 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664552 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664555 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664558 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664560 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664563 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664566 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664569 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664572 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664575 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664578 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664581 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664584 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664587 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664590 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664593 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664595 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664598 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 09:26:58.236315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664600 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664603 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664605 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664608 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664610 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664614 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664617 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664620 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664622 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664625 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664627 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664630 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664632 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664635 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664637 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664640 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664642 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664645 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664647 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 09:26:58.236914 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664651 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664653 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664656 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664660 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664665 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664668 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664671 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664673 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664676 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664679 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664681 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664683 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.664688 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664816 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 09:26:58.237490 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664822 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664825 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664828 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664831 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664834 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664836 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664839 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664841 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664844 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664846 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664849 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664851 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664854 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664856 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664859 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664861 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664864 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664866 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664868 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664871 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 09:26:58.237917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664873 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664876 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664878 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664883 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664887 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664890 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664894 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664898 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664901 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664904 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664907 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664911 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664913 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664916 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664918 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664921 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664924 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664926 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664929 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664932 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 09:26:58.238459 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664934 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664937 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664939 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664942 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664944 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664947 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664949 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664952 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664954 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664957 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664959 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664962 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664964 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664967 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664969 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664975 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664978 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664980 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664983 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664985 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 09:26:58.239030 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664988 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664991 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664994 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.664997 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665000 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665002 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665005 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665007 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665010 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665012 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665015 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665017 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665020 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665023 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665025 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665028 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665030 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665033 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665035 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 09:26:58.239567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665038 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665040 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665043 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665045 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665049 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665053 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665056 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665059 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665062 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665065 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665067 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665070 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665073 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665075 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665078 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665081 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665084 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665087 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 09:26:58.240101 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:56.665089 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.665094 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.666304 2584 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.669411 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.670737 2584 server.go:1019] "Starting client certificate rotation" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.670831 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.670863 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.674149 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.676743 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.689425 2584 log.go:25] "Validated CRI v1 runtime API" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.691475 2584 log.go:25] "Validated CRI v1 image API" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.692237 2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 09:26:58.240614 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.695578 2584 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c2b2224b-d922-4f02-b1d9-010678873baa:/dev/nvme0n1p3 f23685ab-d965-410e-89c6-4a99c65b795f:/dev/nvme0n1p4] Jul 17 09:26:58.240995 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.695596 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 09:26:58.240995 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.699217 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.701671 2584 manager.go:216] Machine: {Timestamp:2026-07-17 09:26:56.70031897 +0000 UTC m=+0.348199573 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090733 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec285656645f143b0ff4ee9dec7243ba SystemUUID:ec285656-645f-143b-0ff4-ee9dec7243ba BootID:0ad3ff7c-1a60-4bef-89fe-3a20a6081cb7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8e:c0:6f:62:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8e:c0:6f:62:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:bb:2e:a1:b3:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.701784 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.701872 2584 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.704735 2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.704761 2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.705465 2584 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.705474 2584 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.705482 2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.706445 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.707887 2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.710550 2584 state_mem.go:36] "Initialized new in-memory state store" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.710667 2584 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.710696 2584 kubelet.go:486] "Attempting to sync node with API server" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.710705 2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.710731 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.710741 2584 kubelet.go:398] "Adding apiserver pod source" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.710749 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 09:26:58.241155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.711123 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.711138 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.714379 2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.716125 2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.716152 2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717368 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717381 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717387 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717393 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717401 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717406 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717445 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717451 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717458 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717464 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717472 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.717503 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.718364 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.718374 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.721397 2584 server.go:1268] "Started kubelet" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.721499 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.721572 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.721582 2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.723025 2584 server.go:310] "Adding debug handlers to kubelet server" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.725895 2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.727141 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d7mp2" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.727161 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d7mp2" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.727849 2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.732127 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.732188 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.732342 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.732944 2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.732958 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.733010 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.732381 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-236.ec2.internal.18c30995e25d0a4d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-236.ec2.internal,UID:ip-10-0-133-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-236.ec2.internal,},FirstTimestamp:2026-07-17 09:26:56.721373773 +0000 UTC m=+0.369254376,LastTimestamp:2026-07-17 09:26:56.721373773 +0000 UTC m=+0.369254376,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-236.ec2.internal,}" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.733435 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.733442 2584 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.733453 2584 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.733558 2584 reconstruct.go:97] "Volume reconstruction finished" Jul 17 09:26:58.241716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.733565 2584 reconciler.go:29] "Reconciler: start to sync state" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.738666 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.738680 2584 factory.go:55] Registering systemd factory Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.738687 2584 factory.go:223] Registration of the systemd container factory successfully Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.738885 2584 factory.go:153] Registering CRI-O factory Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.738900 2584 factory.go:223] Registration of the crio container factory successfully Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.738921 2584 factory.go:103] Registering Raw factory Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.738936 2584 manager.go:1196] Started watching for new ooms in manager Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.739355 2584 manager.go:349] Starting recovery of all containers Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.740962 2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.743548 2584 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-236.ec2.internal\" not found" node="ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.756471 2584 manager.go:354] Recovery completed Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.760563 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.762252 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.762277 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.762288 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.762749 2584 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.762758 2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.762761 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.762778 2584 state_mem.go:36] "Initialized new in-memory state store" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.763382 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.763407 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.763417 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.763923 2584 policy_none.go:49] "None policy: Start" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.763938 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.763948 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.764842 2584 policy_none.go:47] "Start" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.807737 2584 manager.go:337] "Starting Device Plugin manager" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.807768 2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.807779 2584 server.go:87] "Starting device plugin registration server" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.808040 2584 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.808054 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.808131 2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.808223 2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.808233 2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.808705 2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.808782 2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.882556 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.883851 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.883898 2584 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.883927 2584 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:56.883974 2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:56.885853 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.017417 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.019105 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.019140 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.019192 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.019216 2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.026722 2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.242771 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.026878 2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-236.ec2.internal\": node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.039173 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.092912 2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal"] Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.092993 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.094798 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.094826 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.094836 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096055 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096148 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096179 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096759 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096782 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096792 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096791 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096865 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.096878 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.098538 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.098559 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.099202 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientMemory" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.099228 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.099239 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeHasSufficientPID" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.135267 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-236.ec2.internal\" not found" node="ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.139717 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.139799 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-236.ec2.internal\" not found" node="ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.143976 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e350fbe4cd2d7c015dcbec11b3921d8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"6e350fbe4cd2d7c015dcbec11b3921d8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.143996 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e350fbe4cd2d7c015dcbec11b3921d8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"6e350fbe4cd2d7c015dcbec11b3921d8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.144014 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b51292bad5f58bba29f3717a67222f17-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.ec2.internal\" (UID: \"b51292bad5f58bba29f3717a67222f17\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.244341 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.240591 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.411448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.244890 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b51292bad5f58bba29f3717a67222f17-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.ec2.internal\" (UID: \"b51292bad5f58bba29f3717a67222f17\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.411448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.244911 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e350fbe4cd2d7c015dcbec11b3921d8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"6e350fbe4cd2d7c015dcbec11b3921d8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.411448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.244928 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e350fbe4cd2d7c015dcbec11b3921d8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"6e350fbe4cd2d7c015dcbec11b3921d8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.411448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.244953 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6e350fbe4cd2d7c015dcbec11b3921d8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"6e350fbe4cd2d7c015dcbec11b3921d8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.411448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.244988 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b51292bad5f58bba29f3717a67222f17-config\") pod \"kube-apiserver-proxy-ip-10-0-133-236.ec2.internal\" (UID: \"b51292bad5f58bba29f3717a67222f17\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.411448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.245020 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6e350fbe4cd2d7c015dcbec11b3921d8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal\" (UID: \"6e350fbe4cd2d7c015dcbec11b3921d8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.411448 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.341365 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.602899 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.437804 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.602899 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.442487 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.602899 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.443568 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 17 09:26:58.602899 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.543339 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.771114 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.643732 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.771114 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.744221 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.902035 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.778920 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 09:26:58.902035 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.779047 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 09:26:58.902035 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.779075 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 09:26:58.902035 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.840140 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 09:21:57 +0000 UTC" deadline="2027-12-22 10:41:25.31360604 +0000 UTC" Jul 17 09:26:58.902035 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.840171 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12553h14m26.473439333s" Jul 17 09:26:58.902035 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:58.845206 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-236.ec2.internal\" not found" Jul 17 09:26:58.902035 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.851530 2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 09:26:59.067628 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.942060 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" Jul 17 09:26:59.067628 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.952336 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 09:26:59.067628 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.954247 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" Jul 17 09:26:59.067628 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:58.961870 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 09:26:59.257906 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:59.086100 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51292bad5f58bba29f3717a67222f17.slice/crio-6d6e823f85b95cae78684623a90c47b20119b48e1e0d0dd92a8285fee6509696 WatchSource:0}: Error finding container 6d6e823f85b95cae78684623a90c47b20119b48e1e0d0dd92a8285fee6509696: Status 404 returned error can't find the container with id 6d6e823f85b95cae78684623a90c47b20119b48e1e0d0dd92a8285fee6509696 Jul 17 09:26:59.257906 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:26:59.086407 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e350fbe4cd2d7c015dcbec11b3921d8.slice/crio-6bbdbedae809d1dbcbf34f33b5e4278f11a4aaee64125537052e9511eef9c21f WatchSource:0}: Error finding container 6bbdbedae809d1dbcbf34f33b5e4278f11a4aaee64125537052e9511eef9c21f: Status 404 returned error can't find the container with id 6bbdbedae809d1dbcbf34f33b5e4278f11a4aaee64125537052e9511eef9c21f Jul 17 09:26:59.257906 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.092054 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 09:26:59.425555 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.387522 2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 09:26:59.738085 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.737804 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.820216 2584 apiserver.go:52] "Watching apiserver" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.830681 2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.831083 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4b9j2","openshift-multus/multus-additional-cni-plugins-8mqkt","openshift-network-operator/iptables-alerter-m9jpb","openshift-ovn-kubernetes/ovnkube-node-6bwcc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m","openshift-dns/node-resolver-zvws7","openshift-image-registry/node-ca-hkg55","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal","openshift-multus/network-metrics-daemon-vb9ns","openshift-network-diagnostics/network-check-target-s6khr","kube-system/global-pull-secret-syncer-pkvmq","kube-system/konnectivity-agent-sn496","kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bkgq6"] Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.833495 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.835909 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.835972 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.835984 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6tqtg\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.835911 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.836072 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.836330 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.838440 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbtpr\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.838503 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.838764 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.838932 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.840856 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.841058 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.841733 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.841827 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pmkkw\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.841934 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.841981 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.843742 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.843840 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.845146 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.845215 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.845377 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.845436 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.845506 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hsgb5\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.845577 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.846076 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.846365 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zvws7" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.846536 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.846756 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.846979 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-48ck4\"" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.848492 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:26:59.862179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.848878 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.848966 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.849018 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qs7jz\"" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.850546 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.850846 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:59.850915 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.850924 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s4rjt\"" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.851250 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.851250 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.853652 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:59.853709 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.854320 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.856203 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:26:59.856293 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.856912 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-cni-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.856958 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-kubelet\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.856985 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-sys-fs\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857030 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-cni-multus\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857057 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857079 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzscc\" (UniqueName: \"kubernetes.io/projected/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-kube-api-access-hzscc\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857117 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.864228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857139 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-node-log\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857194 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-cni-bin\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857215 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-cni-netd\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857249 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-etc-selinux\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857272 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-cni-bin\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857287 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-os-release\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857303 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857344 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-cnibin\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857365 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-daemon-config\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857387 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-system-cni-dir\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857464 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-os-release\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857502 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-conf-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.857564 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-run-ovn-kubernetes\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858334 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-cni-binary-copy\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858391 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-k8s-cni-cncf-io\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858443 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-ovn\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.867515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858467 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-ovnkube-script-lib\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858530 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-registration-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858571 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858662 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-cni-binary-copy\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858711 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-slash\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858746 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-env-overrides\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858778 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78141cd2-d716-481c-b228-9fdd83ea0b13-iptables-alerter-script\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858806 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858839 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-socket-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858871 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxs6\" (UniqueName: \"kubernetes.io/projected/ff3523e6-3e33-41cc-92c7-8fb082211e91-kube-api-access-jkxs6\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858900 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-kubelet\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858926 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78141cd2-d716-481c-b228-9fdd83ea0b13-host-slash\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858956 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9gd\" (UniqueName: \"kubernetes.io/projected/78141cd2-d716-481c-b228-9fdd83ea0b13-kube-api-access-jk9gd\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.858987 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-var-lib-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859017 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-ovnkube-config\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859089 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5423149f-e3e1-4dae-beab-05130ed98108-ovn-node-metrics-cert\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.868575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859154 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-system-cni-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859184 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-netns\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859224 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-etc-kubernetes\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859253 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-systemd-units\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859286 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-log-socket\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859330 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-run-netns\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859392 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-socket-dir-parent\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859457 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859478 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sn496" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859492 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-systemd\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859616 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-multus-certs\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859649 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-cnibin\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859680 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-etc-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859796 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkg9k\" (UniqueName: \"kubernetes.io/projected/5423149f-e3e1-4dae-beab-05130ed98108-kube-api-access-tkg9k\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859829 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4ssg\" (UniqueName: \"kubernetes.io/projected/ed6be1a8-82df-4378-b517-e3e83bf03857-kube-api-access-q4ssg\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859862 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-hostroot\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.859892 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-device-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.862604 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tz7hj\"" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.862813 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 09:26:59.869374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.862857 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 09:26:59.870155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.863482 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:26:59.870155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.866194 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 09:26:59.870155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.866209 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f88sd\"" Jul 17 09:26:59.870155 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.866282 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.875531 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q2klq" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.884661 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q2klq" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.943303 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960700 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960734 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-multus-certs\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960757 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-cnibin\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960808 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-cnibin\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960806 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-etc-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960847 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkg9k\" (UniqueName: \"kubernetes.io/projected/5423149f-e3e1-4dae-beab-05130ed98108-kube-api-access-tkg9k\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960873 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4ssg\" (UniqueName: \"kubernetes.io/projected/ed6be1a8-82df-4378-b517-e3e83bf03857-kube-api-access-q4ssg\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960882 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-multus-certs\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960899 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-hostroot\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960849 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-etc-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960922 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-device-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960949 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-hostroot\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960951 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-kubelet-config\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.960981 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961006 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-cni-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.995062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961012 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-device-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961034 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-kubelet\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961056 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-sys-fs\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961076 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-run\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961096 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpsj\" (UniqueName: \"kubernetes.io/projected/8c091691-e745-495a-8852-69567f09ce6d-kube-api-access-mtpsj\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961117 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-cni-multus\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961117 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-kubelet\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961165 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961170 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-cni-multus\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961203 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-hosts-file\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961224 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961230 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx4qn\" (UniqueName: \"kubernetes.io/projected/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-kube-api-access-tx4qn\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961202 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-sys-fs\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961233 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-cni-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961272 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-kubernetes\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961301 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzscc\" (UniqueName: \"kubernetes.io/projected/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-kube-api-access-hzscc\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961326 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.997755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961350 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-node-log\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961374 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-cni-bin\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961411 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysctl-d\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961436 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961468 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69fh\" (UniqueName: \"kubernetes.io/projected/62e2a6ea-beae-40ea-a730-c32f3b172707-kube-api-access-j69fh\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961493 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-cni-netd\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961526 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-etc-selinux\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961553 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-cni-bin\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961584 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-os-release\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961608 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961631 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-cnibin\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961658 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-daemon-config\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961681 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-system-cni-dir\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961707 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-os-release\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961730 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-conf-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961755 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-run-ovn-kubernetes\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961793 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-dbus\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:26:59.998726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961831 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-cni-binary-copy\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961855 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-k8s-cni-cncf-io\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961878 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-ovn\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961901 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-ovnkube-script-lib\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961926 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-registration-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961948 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-sys\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961973 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.961995 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-lib-modules\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962018 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c091691-e745-495a-8852-69567f09ce6d-etc-tuned\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962042 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-cni-binary-copy\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962072 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962071 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-slash\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962154 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-env-overrides\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962179 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d99751e-24ae-4349-ab23-b4ce2592a164-konnectivity-ca\") pod \"konnectivity-agent-sn496\" (UID: \"5d99751e-24ae-4349-ab23-b4ce2592a164\") " pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962183 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-cni-netd\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962206 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysctl-conf\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962231 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-var-lib-kubelet\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.000339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962254 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-serviceca\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962279 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78141cd2-d716-481c-b228-9fdd83ea0b13-iptables-alerter-script\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962302 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962321 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-etc-selinux\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962327 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-socket-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962360 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxs6\" (UniqueName: \"kubernetes.io/projected/ff3523e6-3e33-41cc-92c7-8fb082211e91-kube-api-access-jkxs6\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962424 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-socket-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962435 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysconfig\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962461 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c091691-e745-495a-8852-69567f09ce6d-tmp\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962479 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-node-log\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962484 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-host\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962519 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-cni-bin\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962543 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-kubelet\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962543 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-cni-bin\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962125 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-slash\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962621 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-os-release\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962661 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78141cd2-d716-481c-b228-9fdd83ea0b13-host-slash\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:27:00.001338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962686 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9gd\" (UniqueName: \"kubernetes.io/projected/78141cd2-d716-481c-b228-9fdd83ea0b13-kube-api-access-jk9gd\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962725 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-var-lib-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962751 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-host\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962783 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-ovnkube-config\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962809 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5423149f-e3e1-4dae-beab-05130ed98108-ovn-node-metrics-cert\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962834 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-system-cni-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962858 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-netns\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962884 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-etc-kubernetes\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962910 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-systemd-units\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962935 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-log-socket\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962963 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d99751e-24ae-4349-ab23-b4ce2592a164-agent-certs\") pod \"konnectivity-agent-sn496\" (UID: \"5d99751e-24ae-4349-ab23-b4ce2592a164\") " pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962990 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-run-netns\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963029 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-tmp-dir\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963069 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-modprobe-d\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963092 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-systemd\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963118 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbrk6\" (UniqueName: \"kubernetes.io/projected/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-kube-api-access-tbrk6\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963144 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-socket-dir-parent\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.155176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963170 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963198 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-systemd\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.962910 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-env-overrides\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963234 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-etc-kubernetes\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963240 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963300 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-netns\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963306 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-run-ovn-kubernetes\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963339 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78141cd2-d716-481c-b228-9fdd83ea0b13-host-slash\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963350 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-cnibin\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963355 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/78141cd2-d716-481c-b228-9fdd83ea0b13-iptables-alerter-script\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963385 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-conf-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963451 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-socket-dir-parent\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963460 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-var-lib-openvswitch\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963487 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-ovn\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963520 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-system-cni-dir\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963590 2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963601 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963650 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-os-release\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.156578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963821 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-systemd-units\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963875 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-log-socket\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963872 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-run-k8s-cni-cncf-io\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963927 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-run-systemd\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963950 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-cni-binary-copy\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.964072 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-ovnkube-config\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.964096 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-cni-binary-copy\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.964101 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.964134 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed6be1a8-82df-4378-b517-e3e83bf03857-system-cni-dir\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.964148 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5423149f-e3e1-4dae-beab-05130ed98108-host-run-netns\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.964195 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-host-var-lib-kubelet\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.963938 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff3523e6-3e33-41cc-92c7-8fb082211e91-registration-dir\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.964919 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5423149f-e3e1-4dae-beab-05130ed98108-ovnkube-script-lib\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.965082 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-multus-daemon-config\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.965216 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ed6be1a8-82df-4378-b517-e3e83bf03857-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.968081 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5423149f-e3e1-4dae-beab-05130ed98108-ovn-node-metrics-cert\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.969303 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4ssg\" (UniqueName: \"kubernetes.io/projected/ed6be1a8-82df-4378-b517-e3e83bf03857-kube-api-access-q4ssg\") pod \"multus-additional-cni-plugins-8mqkt\" (UID: \"ed6be1a8-82df-4378-b517-e3e83bf03857\") " pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.157515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.970651 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzscc\" (UniqueName: \"kubernetes.io/projected/6063385c-abf2-4e7b-91c2-5bf0d2583bb4-kube-api-access-hzscc\") pod \"multus-4b9j2\" (UID: \"6063385c-abf2-4e7b-91c2-5bf0d2583bb4\") " pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.971527 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxs6\" (UniqueName: \"kubernetes.io/projected/ff3523e6-3e33-41cc-92c7-8fb082211e91-kube-api-access-jkxs6\") pod \"aws-ebs-csi-driver-node-rbx2m\" (UID: \"ff3523e6-3e33-41cc-92c7-8fb082211e91\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.972943 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9gd\" (UniqueName: \"kubernetes.io/projected/78141cd2-d716-481c-b228-9fdd83ea0b13-kube-api-access-jk9gd\") pod \"iptables-alerter-m9jpb\" (UID: \"78141cd2-d716-481c-b228-9fdd83ea0b13\") " pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:26:59.980376 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkg9k\" (UniqueName: \"kubernetes.io/projected/5423149f-e3e1-4dae-beab-05130ed98108-kube-api-access-tkg9k\") pod \"ovnkube-node-6bwcc\" (UID: \"5423149f-e3e1-4dae-beab-05130ed98108\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.002872 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" event={"ID":"b51292bad5f58bba29f3717a67222f17","Type":"ContainerStarted","Data":"6d6e823f85b95cae78684623a90c47b20119b48e1e0d0dd92a8285fee6509696"} Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.004083 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" event={"ID":"6e350fbe4cd2d7c015dcbec11b3921d8","Type":"ContainerStarted","Data":"6bbdbedae809d1dbcbf34f33b5e4278f11a4aaee64125537052e9511eef9c21f"} Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064087 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-dbus\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064127 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-sys\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064154 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-lib-modules\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064172 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c091691-e745-495a-8852-69567f09ce6d-etc-tuned\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064194 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d99751e-24ae-4349-ab23-b4ce2592a164-konnectivity-ca\") pod \"konnectivity-agent-sn496\" (UID: \"5d99751e-24ae-4349-ab23-b4ce2592a164\") " pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064218 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysctl-conf\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064229 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-dbus\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064296 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-lib-modules\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064296 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-sys\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064237 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-var-lib-kubelet\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064340 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-var-lib-kubelet\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064364 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-serviceca\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.158502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064370 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysctl-conf\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064408 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysconfig\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064434 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c091691-e745-495a-8852-69567f09ce6d-tmp\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064455 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-host\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064480 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-host\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064508 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d99751e-24ae-4349-ab23-b4ce2592a164-agent-certs\") pod \"konnectivity-agent-sn496\" (UID: \"5d99751e-24ae-4349-ab23-b4ce2592a164\") " pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064531 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-tmp-dir\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064539 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysconfig\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064556 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-modprobe-d\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064599 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-host\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064628 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-systemd\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064651 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-modprobe-d\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064656 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbrk6\" (UniqueName: \"kubernetes.io/projected/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-kube-api-access-tbrk6\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064688 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064719 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-kubelet-config\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064744 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064770 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-run\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064776 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d99751e-24ae-4349-ab23-b4ce2592a164-konnectivity-ca\") pod \"konnectivity-agent-sn496\" (UID: \"5d99751e-24ae-4349-ab23-b4ce2592a164\") " pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:00.159332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064792 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpsj\" (UniqueName: \"kubernetes.io/projected/8c091691-e745-495a-8852-69567f09ce6d-kube-api-access-mtpsj\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064810 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-serviceca\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064824 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-hosts-file\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064859 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx4qn\" (UniqueName: \"kubernetes.io/projected/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-kube-api-access-tx4qn\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064881 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-kubernetes\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.064904 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064928 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-tmp-dir\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064977 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-host\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.064995 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret podName:ff2f29e6-ee5e-433c-a625-30090f8f5d9c nodeName:}" failed. No retries permitted until 2026-07-17 09:27:00.564954499 +0000 UTC m=+3.104153446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret") pod "global-pull-secret-syncer-pkvmq" (UID: "ff2f29e6-ee5e-433c-a625-30090f8f5d9c") : object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.065009 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysctl-d\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.065041 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-run\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.065059 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-kubernetes\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.065108 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-systemd\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.064906 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8c091691-e745-495a-8852-69567f09ce6d-etc-sysctl-d\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.065162 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.065196 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69fh\" (UniqueName: \"kubernetes.io/projected/62e2a6ea-beae-40ea-a730-c32f3b172707-kube-api-access-j69fh\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.065223 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-kubelet-config\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:00.160275 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.065294 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.065345 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs podName:62e2a6ea-beae-40ea-a730-c32f3b172707 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:00.565330948 +0000 UTC m=+3.104529892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs") pod "network-metrics-daemon-vb9ns" (UID: "62e2a6ea-beae-40ea-a730-c32f3b172707") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.065370 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-hosts-file\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.067503 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8c091691-e745-495a-8852-69567f09ce6d-etc-tuned\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.067815 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c091691-e745-495a-8852-69567f09ce6d-tmp\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.068712 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d99751e-24ae-4349-ab23-b4ce2592a164-agent-certs\") pod \"konnectivity-agent-sn496\" (UID: \"5d99751e-24ae-4349-ab23-b4ce2592a164\") " pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.071039 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.071061 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.071075 2584 projected.go:196] Error preparing data for projected volume kube-api-access-t9st6 for pod openshift-network-diagnostics/network-check-target-s6khr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.071126 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6 podName:1d34fb73-3a61-49ce-be61-32b998fa34f1 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:00.571110543 +0000 UTC m=+3.110309492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t9st6" (UniqueName: "kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6") pod "network-check-target-s6khr" (UID: "1d34fb73-3a61-49ce-be61-32b998fa34f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.073414 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpsj\" (UniqueName: \"kubernetes.io/projected/8c091691-e745-495a-8852-69567f09ce6d-kube-api-access-mtpsj\") pod \"tuned-bkgq6\" (UID: \"8c091691-e745-495a-8852-69567f09ce6d\") " pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.073482 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx4qn\" (UniqueName: \"kubernetes.io/projected/eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7-kube-api-access-tx4qn\") pod \"node-resolver-zvws7\" (UID: \"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7\") " pod="openshift-dns/node-resolver-zvws7" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.073869 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69fh\" (UniqueName: \"kubernetes.io/projected/62e2a6ea-beae-40ea-a730-c32f3b172707-kube-api-access-j69fh\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.080799 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbrk6\" (UniqueName: \"kubernetes.io/projected/5ab356ac-a51c-44dc-b850-1a0f814a0b0f-kube-api-access-tbrk6\") pod \"node-ca-hkg55\" (UID: \"5ab356ac-a51c-44dc-b850-1a0f814a0b0f\") " pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.145821 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4b9j2" Jul 17 09:27:00.161154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.153538 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" Jul 17 09:27:00.173130 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.173105 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m9jpb" Jul 17 09:27:00.187189 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.187165 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:00.193968 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.193946 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" Jul 17 09:27:00.204693 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.204628 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zvws7" Jul 17 09:27:00.218408 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.218336 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hkg55" Jul 17 09:27:00.226234 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.226208 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:00.231961 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.231940 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" Jul 17 09:27:00.344081 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.344051 2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 09:27:00.569212 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.569116 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:00.569212 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.569163 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:00.569433 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.569294 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:00.569433 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.569358 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret podName:ff2f29e6-ee5e-433c-a625-30090f8f5d9c nodeName:}" failed. No retries permitted until 2026-07-17 09:27:01.569342792 +0000 UTC m=+4.108541736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret") pod "global-pull-secret-syncer-pkvmq" (UID: "ff2f29e6-ee5e-433c-a625-30090f8f5d9c") : object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:00.569433 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.569294 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:00.569562 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.569465 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs podName:62e2a6ea-beae-40ea-a730-c32f3b172707 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:01.569446271 +0000 UTC m=+4.108645230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs") pod "network-metrics-daemon-vb9ns" (UID: "62e2a6ea-beae-40ea-a730-c32f3b172707") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:00.670181 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.670141 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:00.670349 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.670279 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 09:27:00.670349 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.670307 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 09:27:00.670349 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.670320 2584 projected.go:196] Error preparing data for projected volume kube-api-access-t9st6 for pod openshift-network-diagnostics/network-check-target-s6khr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:00.670486 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.670389 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6 podName:1d34fb73-3a61-49ce-be61-32b998fa34f1 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:01.670369437 +0000 UTC m=+4.209568400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9st6" (UniqueName: "kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6") pod "network-check-target-s6khr" (UID: "1d34fb73-3a61-49ce-be61-32b998fa34f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:00.888063 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.887968 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 09:21:59 +0000 UTC" deadline="2028-01-21 15:00:57.613849717 +0000 UTC" Jul 17 09:27:00.888063 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.888012 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13277h33m56.725842567s" Jul 17 09:27:00.993708 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.993676 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:00.993868 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:00.993682 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:00.993868 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.993794 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:00.993995 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:00.993890 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:01.121534 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.121506 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6be1a8_82df_4378_b517_e3e83bf03857.slice/crio-6e35876dae924c2226b7d74ad3b21c9953270ab566372ee957c4e2da7cfa5c13 WatchSource:0}: Error finding container 6e35876dae924c2226b7d74ad3b21c9953270ab566372ee957c4e2da7cfa5c13: Status 404 returned error can't find the container with id 6e35876dae924c2226b7d74ad3b21c9953270ab566372ee957c4e2da7cfa5c13 Jul 17 09:27:01.123218 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.123194 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3523e6_3e33_41cc_92c7_8fb082211e91.slice/crio-91714ee0aeef6aeebda30b1e042208e9fad71aebc8600f668a2bfd00ed69b274 WatchSource:0}: Error finding container 91714ee0aeef6aeebda30b1e042208e9fad71aebc8600f668a2bfd00ed69b274: Status 404 returned error can't find the container with id 91714ee0aeef6aeebda30b1e042208e9fad71aebc8600f668a2bfd00ed69b274 Jul 17 09:27:01.126352 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.126331 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5423149f_e3e1_4dae_beab_05130ed98108.slice/crio-1371f9de1f57124d83c1ec6bc23bfc2fc7e4b1e40abe48fc84eccec8b2529d69 WatchSource:0}: Error finding container 1371f9de1f57124d83c1ec6bc23bfc2fc7e4b1e40abe48fc84eccec8b2529d69: Status 404 returned error can't find the container with id 1371f9de1f57124d83c1ec6bc23bfc2fc7e4b1e40abe48fc84eccec8b2529d69 Jul 17 09:27:01.127443 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.127419 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c091691_e745_495a_8852_69567f09ce6d.slice/crio-7fa1a9ec61ea10c41d58c02d41bfeef40131cfa3eb33d68ffed5833b20137d7f WatchSource:0}: Error finding container 7fa1a9ec61ea10c41d58c02d41bfeef40131cfa3eb33d68ffed5833b20137d7f: Status 404 returned error can't find the container with id 7fa1a9ec61ea10c41d58c02d41bfeef40131cfa3eb33d68ffed5833b20137d7f Jul 17 09:27:01.128225 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.128205 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab356ac_a51c_44dc_b850_1a0f814a0b0f.slice/crio-ab85a3a156ee09eac242048c2350aca59a51a589cef5f070e377d78023db1cf5 WatchSource:0}: Error finding container ab85a3a156ee09eac242048c2350aca59a51a589cef5f070e377d78023db1cf5: Status 404 returned error can't find the container with id ab85a3a156ee09eac242048c2350aca59a51a589cef5f070e377d78023db1cf5 Jul 17 09:27:01.128687 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.128666 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6063385c_abf2_4e7b_91c2_5bf0d2583bb4.slice/crio-b8c15d2dd307963730cbd3abf69bb4eabcd0b2decba77518ad0143c767394748 WatchSource:0}: Error finding container b8c15d2dd307963730cbd3abf69bb4eabcd0b2decba77518ad0143c767394748: Status 404 returned error can't find the container with id b8c15d2dd307963730cbd3abf69bb4eabcd0b2decba77518ad0143c767394748 Jul 17 09:27:01.130843 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.130816 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78141cd2_d716_481c_b228_9fdd83ea0b13.slice/crio-95b2a16d45d5647f636542ea47baa15c3f23ff04b9c9a51332d70ed2d8b63e41 WatchSource:0}: Error finding container 95b2a16d45d5647f636542ea47baa15c3f23ff04b9c9a51332d70ed2d8b63e41: Status 404 returned error can't find the container with id 95b2a16d45d5647f636542ea47baa15c3f23ff04b9c9a51332d70ed2d8b63e41 Jul 17 09:27:01.132567 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.131992 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d99751e_24ae_4349_ab23_b4ce2592a164.slice/crio-4679bcef297a8b37200aa4e5bec11571ee37b8f570bbd0803f9afbacc14df4a1 WatchSource:0}: Error finding container 4679bcef297a8b37200aa4e5bec11571ee37b8f570bbd0803f9afbacc14df4a1: Status 404 returned error can't find the container with id 4679bcef297a8b37200aa4e5bec11571ee37b8f570bbd0803f9afbacc14df4a1 Jul 17 09:27:01.133273 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:01.133244 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb10dd55_281e_4ebf_ac8f_d79c9e6e55e7.slice/crio-c15f873e666cfb3989089bb5d1a2016902f64582b343334a5f637100b37e2f56 WatchSource:0}: Error finding container c15f873e666cfb3989089bb5d1a2016902f64582b343334a5f637100b37e2f56: Status 404 returned error can't find the container with id c15f873e666cfb3989089bb5d1a2016902f64582b343334a5f637100b37e2f56 Jul 17 09:27:01.578132 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:01.577777 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:01.578132 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.578072 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:01.578132 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:01.578096 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:01.578387 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.578148 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret podName:ff2f29e6-ee5e-433c-a625-30090f8f5d9c nodeName:}" failed. No retries permitted until 2026-07-17 09:27:03.578127814 +0000 UTC m=+6.117326769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret") pod "global-pull-secret-syncer-pkvmq" (UID: "ff2f29e6-ee5e-433c-a625-30090f8f5d9c") : object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:01.578387 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.578208 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:01.578387 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.578258 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs podName:62e2a6ea-beae-40ea-a730-c32f3b172707 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:03.578243136 +0000 UTC m=+6.117442093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs") pod "network-metrics-daemon-vb9ns" (UID: "62e2a6ea-beae-40ea-a730-c32f3b172707") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:01.679524 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:01.679486 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:01.679716 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.679692 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 09:27:01.679796 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.679723 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 09:27:01.679796 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.679743 2584 projected.go:196] Error preparing data for projected volume kube-api-access-t9st6 for pod openshift-network-diagnostics/network-check-target-s6khr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:01.679899 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.679803 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6 podName:1d34fb73-3a61-49ce-be61-32b998fa34f1 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:03.679784742 +0000 UTC m=+6.218983706 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9st6" (UniqueName: "kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6") pod "network-check-target-s6khr" (UID: "1d34fb73-3a61-49ce-be61-32b998fa34f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:01.888218 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:01.888130 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 09:21:59 +0000 UTC" deadline="2028-03-02 07:07:40.131234062 +0000 UTC" Jul 17 09:27:01.888218 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:01.888174 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14253h40m38.243064433s" Jul 17 09:27:01.994516 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:01.994031 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:01.994516 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:01.994160 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:02.018454 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.017287 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" event={"ID":"8c091691-e745-495a-8852-69567f09ce6d","Type":"ContainerStarted","Data":"7fa1a9ec61ea10c41d58c02d41bfeef40131cfa3eb33d68ffed5833b20137d7f"} Jul 17 09:27:02.019664 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.019638 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" event={"ID":"ff3523e6-3e33-41cc-92c7-8fb082211e91","Type":"ContainerStarted","Data":"91714ee0aeef6aeebda30b1e042208e9fad71aebc8600f668a2bfd00ed69b274"} Jul 17 09:27:02.022485 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.022459 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" event={"ID":"ed6be1a8-82df-4378-b517-e3e83bf03857","Type":"ContainerStarted","Data":"6e35876dae924c2226b7d74ad3b21c9953270ab566372ee957c4e2da7cfa5c13"} Jul 17 09:27:02.027918 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.027889 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" event={"ID":"b51292bad5f58bba29f3717a67222f17","Type":"ContainerStarted","Data":"3bb4e8e84abbe68b788bc58e242dc11c9789852841e7c545706efa7aabddc883"} Jul 17 09:27:02.037099 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.037075 2584 generic.go:356] "Generic (PLEG): container finished" podID="6e350fbe4cd2d7c015dcbec11b3921d8" containerID="2823700dfe00990dd9795fe0cec70e51f31e322e5f7e3eb454bd1d7b2ef515d7" exitCode=0 Jul 17 09:27:02.037220 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.037147 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" event={"ID":"6e350fbe4cd2d7c015dcbec11b3921d8","Type":"ContainerDied","Data":"2823700dfe00990dd9795fe0cec70e51f31e322e5f7e3eb454bd1d7b2ef515d7"} Jul 17 09:27:02.045973 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.045949 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sn496" event={"ID":"5d99751e-24ae-4349-ab23-b4ce2592a164","Type":"ContainerStarted","Data":"4679bcef297a8b37200aa4e5bec11571ee37b8f570bbd0803f9afbacc14df4a1"} Jul 17 09:27:02.050012 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.049971 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m9jpb" event={"ID":"78141cd2-d716-481c-b228-9fdd83ea0b13","Type":"ContainerStarted","Data":"95b2a16d45d5647f636542ea47baa15c3f23ff04b9c9a51332d70ed2d8b63e41"} Jul 17 09:27:02.056231 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.056180 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-236.ec2.internal" podStartSLOduration=4.056162493 podStartE2EDuration="4.056162493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:02.04247494 +0000 UTC m=+4.581673915" watchObservedRunningTime="2026-07-17 09:27:02.056162493 +0000 UTC m=+4.595361458" Jul 17 09:27:02.091927 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.091894 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zvws7" event={"ID":"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7","Type":"ContainerStarted","Data":"c15f873e666cfb3989089bb5d1a2016902f64582b343334a5f637100b37e2f56"} Jul 17 09:27:02.100879 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.100847 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4b9j2" event={"ID":"6063385c-abf2-4e7b-91c2-5bf0d2583bb4","Type":"ContainerStarted","Data":"b8c15d2dd307963730cbd3abf69bb4eabcd0b2decba77518ad0143c767394748"} Jul 17 09:27:02.114798 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.114766 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"1371f9de1f57124d83c1ec6bc23bfc2fc7e4b1e40abe48fc84eccec8b2529d69"} Jul 17 09:27:02.136622 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.136530 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hkg55" event={"ID":"5ab356ac-a51c-44dc-b850-1a0f814a0b0f","Type":"ContainerStarted","Data":"ab85a3a156ee09eac242048c2350aca59a51a589cef5f070e377d78023db1cf5"} Jul 17 09:27:02.993980 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.993945 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:02.994146 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:02.994068 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:02.994486 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:02.994466 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:02.994606 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:02.994569 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:03.155527 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:03.155472 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" event={"ID":"6e350fbe4cd2d7c015dcbec11b3921d8","Type":"ContainerStarted","Data":"b703d494f02c6680ca9a06170d9a0807303ebc05b817414526199d0c50b80609"} Jul 17 09:27:03.598913 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:03.598874 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:03.599079 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:03.598937 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:03.599153 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.599096 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:03.599204 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.599160 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs podName:62e2a6ea-beae-40ea-a730-c32f3b172707 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:07.599140637 +0000 UTC m=+10.138339593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs") pod "network-metrics-daemon-vb9ns" (UID: "62e2a6ea-beae-40ea-a730-c32f3b172707") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:03.599607 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.599583 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:03.599723 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.599642 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret podName:ff2f29e6-ee5e-433c-a625-30090f8f5d9c nodeName:}" failed. No retries permitted until 2026-07-17 09:27:07.599625476 +0000 UTC m=+10.138824424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret") pod "global-pull-secret-syncer-pkvmq" (UID: "ff2f29e6-ee5e-433c-a625-30090f8f5d9c") : object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:03.699818 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:03.699781 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:03.700564 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.700026 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 09:27:03.700564 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.700050 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 09:27:03.700564 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.700063 2584 projected.go:196] Error preparing data for projected volume kube-api-access-t9st6 for pod openshift-network-diagnostics/network-check-target-s6khr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:03.700564 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.700122 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6 podName:1d34fb73-3a61-49ce-be61-32b998fa34f1 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:07.700103784 +0000 UTC m=+10.239302752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9st6" (UniqueName: "kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6") pod "network-check-target-s6khr" (UID: "1d34fb73-3a61-49ce-be61-32b998fa34f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:03.995669 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:03.995591 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:03.995816 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:03.995716 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:04.994081 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:04.994045 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:04.994564 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:04.994178 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:04.994564 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:04.994553 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:04.994684 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:04.994650 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:05.993731 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:05.993547 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:05.993731 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:05.993689 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:06.993956 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:06.993852 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:06.994447 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:06.993973 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:06.994447 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:06.994009 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:06.994447 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:06.994091 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:07.635493 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:07.634763 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:07.635493 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:07.634821 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:07.635493 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.634969 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:07.635493 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.635030 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs podName:62e2a6ea-beae-40ea-a730-c32f3b172707 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:15.635010857 +0000 UTC m=+18.174209806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs") pod "network-metrics-daemon-vb9ns" (UID: "62e2a6ea-beae-40ea-a730-c32f3b172707") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:07.635493 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.635392 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:07.635493 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.635462 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret podName:ff2f29e6-ee5e-433c-a625-30090f8f5d9c nodeName:}" failed. No retries permitted until 2026-07-17 09:27:15.635446827 +0000 UTC m=+18.174645778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret") pod "global-pull-secret-syncer-pkvmq" (UID: "ff2f29e6-ee5e-433c-a625-30090f8f5d9c") : object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:07.735508 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:07.735279 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:07.735670 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.735521 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 09:27:07.735670 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.735546 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 09:27:07.735670 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.735558 2584 projected.go:196] Error preparing data for projected volume kube-api-access-t9st6 for pod openshift-network-diagnostics/network-check-target-s6khr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:07.735670 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.735617 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6 podName:1d34fb73-3a61-49ce-be61-32b998fa34f1 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:15.735600918 +0000 UTC m=+18.274799864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9st6" (UniqueName: "kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6") pod "network-check-target-s6khr" (UID: "1d34fb73-3a61-49ce-be61-32b998fa34f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:07.993981 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:07.993786 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:07.993981 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:07.993901 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:08.993237 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:08.993197 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:08.993445 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:08.993335 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:08.993654 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:08.993631 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:08.993793 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:08.993756 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:09.993007 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:09.992970 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:09.993496 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:09.993110 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:10.993426 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:10.993377 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:10.993874 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:10.993381 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:10.993874 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:10.993496 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:10.993874 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:10.993599 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:11.992918 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:11.992840 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:11.993068 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:11.992980 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:12.993221 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:12.993187 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:12.993221 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:12.993210 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:12.993736 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:12.993308 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:12.993736 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:12.993461 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:13.993725 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:13.993692 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:13.994159 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:13.993830 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:14.993716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:14.993676 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:14.993716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:14.993701 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:14.994302 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:14.993798 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:14.994302 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:14.993896 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:15.700302 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:15.700273 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:15.700302 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:15.700312 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:15.700498 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.700437 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:15.700498 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.700497 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret podName:ff2f29e6-ee5e-433c-a625-30090f8f5d9c nodeName:}" failed. No retries permitted until 2026-07-17 09:27:31.70048271 +0000 UTC m=+34.239681659 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret") pod "global-pull-secret-syncer-pkvmq" (UID: "ff2f29e6-ee5e-433c-a625-30090f8f5d9c") : object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:15.700581 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.700437 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:15.700581 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.700561 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs podName:62e2a6ea-beae-40ea-a730-c32f3b172707 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:31.70054865 +0000 UTC m=+34.239747608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs") pod "network-metrics-daemon-vb9ns" (UID: "62e2a6ea-beae-40ea-a730-c32f3b172707") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:15.801577 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:15.801533 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:15.801729 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.801702 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 09:27:15.801729 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.801729 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 09:27:15.801839 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.801743 2584 projected.go:196] Error preparing data for projected volume kube-api-access-t9st6 for pod openshift-network-diagnostics/network-check-target-s6khr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:15.801839 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.801802 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6 podName:1d34fb73-3a61-49ce-be61-32b998fa34f1 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:31.80178731 +0000 UTC m=+34.340986258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9st6" (UniqueName: "kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6") pod "network-check-target-s6khr" (UID: "1d34fb73-3a61-49ce-be61-32b998fa34f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:15.993478 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:15.993379 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:15.993624 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:15.993532 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:16.992813 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:16.992780 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:16.993293 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:16.992780 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:16.993293 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:16.992926 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:16.993293 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:16.992993 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:17.994182 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:17.994150 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:17.994603 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:17.994257 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:18.993128 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:18.993090 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:18.993304 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:18.993207 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:18.993304 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:18.993267 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:18.993419 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:18.993361 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:19.993478 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:19.993065 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:19.993478 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:19.993384 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:20.193886 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.193653 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4b9j2" event={"ID":"6063385c-abf2-4e7b-91c2-5bf0d2583bb4","Type":"ContainerStarted","Data":"491bc5745bf933c0e1aa3812d01adc28b536142cfe096b5f6114dfcb020c5ecb"} Jul 17 09:27:20.195814 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.195787 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"d1cf9197994a4fe0732721914ffae1ae670af84ae19369ec179b7b1211e48b7a"} Jul 17 09:27:20.195931 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.195871 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"592eb83e5c0c54e3476db793d4845868799cc3d936df3419a581b9f5372f46cf"} Jul 17 09:27:20.195931 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.195915 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"7d8eeb0a8d7c0d7de64ae92141cb1b1243695f3cc4652e7cf5868f83c478134e"} Jul 17 09:27:20.197053 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.197031 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hkg55" event={"ID":"5ab356ac-a51c-44dc-b850-1a0f814a0b0f","Type":"ContainerStarted","Data":"94833eba0a1458321999dd59b488a84d52805bffa4ef612ef33c8491ea454bf0"} Jul 17 09:27:20.198451 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.198428 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" event={"ID":"8c091691-e745-495a-8852-69567f09ce6d","Type":"ContainerStarted","Data":"4f03a11d050f972c4e0f0c4df45478a10ff1c7ace2a87b599ee67375da85338f"} Jul 17 09:27:20.199914 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.199892 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" event={"ID":"ff3523e6-3e33-41cc-92c7-8fb082211e91","Type":"ContainerStarted","Data":"2adf9d80ccb0c837e0dce2c31b5dfecc6dd7f04eeedff8ee966fd16797fe7be2"} Jul 17 09:27:20.204125 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.201827 2584 generic.go:356] "Generic (PLEG): container finished" podID="ed6be1a8-82df-4378-b517-e3e83bf03857" containerID="961850fb1200ad32e613be906b3494dd3bd9428d3adfd15c52291387e3cc26c1" exitCode=0 Jul 17 09:27:20.204125 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.202181 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" event={"ID":"ed6be1a8-82df-4378-b517-e3e83bf03857","Type":"ContainerDied","Data":"961850fb1200ad32e613be906b3494dd3bd9428d3adfd15c52291387e3cc26c1"} Jul 17 09:27:20.205425 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.205310 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sn496" event={"ID":"5d99751e-24ae-4349-ab23-b4ce2592a164","Type":"ContainerStarted","Data":"5d617fe20212a3997120f01745e4c0206215843a80c9cdb9c68842798032faf6"} Jul 17 09:27:20.206818 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.206799 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zvws7" event={"ID":"eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7","Type":"ContainerStarted","Data":"fb7cadaa215e7086deddf35fd3aaff4dc0c15051e0c793368b1ca94c08bfa80f"} Jul 17 09:27:20.212025 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.211990 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-236.ec2.internal" podStartSLOduration=22.211980194 podStartE2EDuration="22.211980194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:03.169878567 +0000 UTC m=+5.709077535" watchObservedRunningTime="2026-07-17 09:27:20.211980194 +0000 UTC m=+22.751179159" Jul 17 09:27:20.212236 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.212219 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4b9j2" podStartSLOduration=3.477609643 podStartE2EDuration="22.212213866s" totalImagesPullingTime="18.734604223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:20.211392074 +0000 UTC m=+22.750591040" watchObservedRunningTime="2026-07-17 09:27:20.212213866 +0000 UTC m=+22.751412831" Jul 17 09:27:20.242725 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.242685 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hkg55" podStartSLOduration=3.561927749 podStartE2EDuration="22.242669648s" totalImagesPullingTime="18.680741899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:20.242064583 +0000 UTC m=+22.781263550" watchObservedRunningTime="2026-07-17 09:27:20.242669648 +0000 UTC m=+22.781868613" Jul 17 09:27:20.259265 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.259222 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sn496" podStartSLOduration=11.643519386 podStartE2EDuration="22.259208866s" totalImagesPullingTime="10.61568948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:20.258985057 +0000 UTC m=+22.798184027" watchObservedRunningTime="2026-07-17 09:27:20.259208866 +0000 UTC m=+22.798407829" Jul 17 09:27:20.274669 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.274622 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bkgq6" podStartSLOduration=3.590540722 podStartE2EDuration="22.274609418s" totalImagesPullingTime="18.684068696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:20.274577378 +0000 UTC m=+22.813776345" watchObservedRunningTime="2026-07-17 09:27:20.274609418 +0000 UTC m=+22.813808381" Jul 17 09:27:20.290305 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.290267 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zvws7" podStartSLOduration=3.615766409 podStartE2EDuration="22.290255672s" totalImagesPullingTime="18.674489263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:20.289898935 +0000 UTC m=+22.829097915" watchObservedRunningTime="2026-07-17 09:27:20.290255672 +0000 UTC m=+22.829454638" Jul 17 09:27:20.993245 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.993216 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:20.993386 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:20.993216 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:20.993386 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:20.993321 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:20.993483 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:20.993392 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:21.210109 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.210068 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m9jpb" event={"ID":"78141cd2-d716-481c-b228-9fdd83ea0b13","Type":"ContainerStarted","Data":"40eb3692252abed33cc54e3228578f2b34ea9dea82fe3fe9d9e03b7b5c720230"} Jul 17 09:27:21.213598 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.213523 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"7d01bcfdd47693b51dec70abaf73e02f302d07f27be37347ba1f389bd80cb883"} Jul 17 09:27:21.213723 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.213678 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"daa45ff46565dcd585ed99b7457f5a49568d89c46569cc4cfdf4d2a359fb930f"} Jul 17 09:27:21.213790 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.213769 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"5dd56aea01a5a8e0dc72a4fe5442201da02e4b2f34bb990ee4577061891cbe61"} Jul 17 09:27:21.222958 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.222912 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m9jpb" podStartSLOduration=4.544075636 podStartE2EDuration="23.222898967s" totalImagesPullingTime="18.678823331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:21.222881144 +0000 UTC m=+23.762080112" watchObservedRunningTime="2026-07-17 09:27:21.222898967 +0000 UTC m=+23.762097933" Jul 17 09:27:21.449550 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.449520 2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 09:27:21.850949 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.850917 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:21.851571 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.851554 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:21.927465 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.927334 2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T09:27:21.449546242Z","UUID":"87fe0ad6-5993-4c38-b96e-c6dbd612f357","Handler":null,"Name":"","Endpoint":""} Jul 17 09:27:21.929082 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.929057 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 09:27:21.929202 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.929093 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 09:27:21.993612 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:21.993588 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:21.994098 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:21.993706 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:22.218573 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:22.218495 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" event={"ID":"ff3523e6-3e33-41cc-92c7-8fb082211e91","Type":"ContainerStarted","Data":"8c2b6514d75f37806c50279999a7e2a88ab3410d6ff149f9081b638944bec14c"} Jul 17 09:27:22.993611 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:22.993540 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:22.993760 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:22.993654 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:22.993760 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:22.993709 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:22.994115 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:22.993809 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:23.224547 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:23.224512 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"a306163f0105a08c5b13cf7efee5a2d4a4e9295e2a2a3e25ac03a9f23ad51e6c"} Jul 17 09:27:23.226784 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:23.226753 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" event={"ID":"ff3523e6-3e33-41cc-92c7-8fb082211e91","Type":"ContainerStarted","Data":"2d975b5b8849c027d60499bf81274ecf0a5bf3e9b420a02c621d3c0243c83dc6"} Jul 17 09:27:23.226908 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:23.226786 2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 09:27:23.244994 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:23.244906 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rbx2m" podStartSLOduration=3.997690042 podStartE2EDuration="25.24488816s" totalImagesPullingTime="21.247198118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:23.243262549 +0000 UTC m=+25.782461537" watchObservedRunningTime="2026-07-17 09:27:23.24488816 +0000 UTC m=+25.784087127" Jul 17 09:27:23.993789 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:23.993755 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:23.994198 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:23.993865 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:24.993793 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:24.993602 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:24.994215 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:24.993665 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:24.994215 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:24.993903 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:24.994215 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:24.993977 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:25.235691 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:25.235650 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" event={"ID":"5423149f-e3e1-4dae-beab-05130ed98108","Type":"ContainerStarted","Data":"58773e14f8cf091ee9a0148d0574b014459d17d73114348555034d46c697936a"} Jul 17 09:27:25.236078 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:25.236059 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:25.236173 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:25.236140 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:25.236227 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:25.236190 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:25.252608 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:25.252578 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:25.252739 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:25.252726 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:25.260870 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:25.260830 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" podStartSLOduration=8.511973409 podStartE2EDuration="27.260817326s" totalImagesPullingTime="18.748843917s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:25.259158493 +0000 UTC m=+27.798357459" watchObservedRunningTime="2026-07-17 09:27:25.260817326 +0000 UTC m=+27.800016314" Jul 17 09:27:25.993657 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:25.993391 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:25.993827 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:25.993772 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:26.239763 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:26.239728 2584 generic.go:356] "Generic (PLEG): container finished" podID="ed6be1a8-82df-4378-b517-e3e83bf03857" containerID="a6c0c6ecf1fd33f2f09cc7cb4e00b3a864fd7e7204eb36a87080818deea7116b" exitCode=0 Jul 17 09:27:26.239955 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:26.239790 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" event={"ID":"ed6be1a8-82df-4378-b517-e3e83bf03857","Type":"ContainerDied","Data":"a6c0c6ecf1fd33f2f09cc7cb4e00b3a864fd7e7204eb36a87080818deea7116b"} Jul 17 09:27:26.993842 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:26.993807 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:26.994429 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:26.993819 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:26.994429 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:26.993942 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:26.994429 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:26.993994 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:27.243902 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:27.243832 2584 generic.go:356] "Generic (PLEG): container finished" podID="ed6be1a8-82df-4378-b517-e3e83bf03857" containerID="73eecb84cdf60556eb249598358258ae77f7d53fa04f4043981bd92cc511bfe4" exitCode=0 Jul 17 09:27:27.244025 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:27.243914 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" event={"ID":"ed6be1a8-82df-4378-b517-e3e83bf03857","Type":"ContainerDied","Data":"73eecb84cdf60556eb249598358258ae77f7d53fa04f4043981bd92cc511bfe4"} Jul 17 09:27:27.345868 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:27.345836 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vb9ns"] Jul 17 09:27:27.346013 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:27.345936 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:27.346062 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:27.346024 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:27.348735 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:27.348711 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s6khr"] Jul 17 09:27:27.348851 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:27.348803 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:27.348910 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:27.348894 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:27.358972 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:27.358940 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pkvmq"] Jul 17 09:27:27.359080 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:27.359032 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:27.359133 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:27.359110 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:28.343435 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:28.343404 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:28.343809 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:28.343574 2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 09:27:28.343951 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:28.343930 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sn496" Jul 17 09:27:28.993179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:28.992992 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:28.993360 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:28.992996 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:28.993360 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:28.993259 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:28.993360 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:28.993039 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:28.993535 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:28.993373 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:28.993535 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:28.993459 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:29.251342 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:29.251260 2584 generic.go:356] "Generic (PLEG): container finished" podID="ed6be1a8-82df-4378-b517-e3e83bf03857" containerID="0fe70093c1077dc9ed01c75c0fa076c867effb2c26d269ad4b458a85327476fc" exitCode=0 Jul 17 09:27:29.251342 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:29.251297 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" event={"ID":"ed6be1a8-82df-4378-b517-e3e83bf03857","Type":"ContainerDied","Data":"0fe70093c1077dc9ed01c75c0fa076c867effb2c26d269ad4b458a85327476fc"} Jul 17 09:27:30.993015 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:30.992977 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:30.993640 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:30.993035 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:30.993640 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:30.992992 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:30.993640 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:30.993117 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:30.993640 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:30.993231 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:30.993640 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:30.993311 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:31.718814 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:31.718733 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:31.718814 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:31.718787 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:31.719011 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:31.718893 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:31.719011 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:31.718895 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:31.719011 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:31.718956 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs podName:62e2a6ea-beae-40ea-a730-c32f3b172707 nodeName:}" failed. No retries permitted until 2026-07-17 09:28:03.71893698 +0000 UTC m=+66.258135930 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs") pod "network-metrics-daemon-vb9ns" (UID: "62e2a6ea-beae-40ea-a730-c32f3b172707") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 09:27:31.719011 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:31.718972 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret podName:ff2f29e6-ee5e-433c-a625-30090f8f5d9c nodeName:}" failed. No retries permitted until 2026-07-17 09:28:03.718965877 +0000 UTC m=+66.258164821 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret") pod "global-pull-secret-syncer-pkvmq" (UID: "ff2f29e6-ee5e-433c-a625-30090f8f5d9c") : object "kube-system"/"original-pull-secret" not registered Jul 17 09:27:31.819738 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:31.819704 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:31.819913 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:31.819831 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 09:27:31.819913 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:31.819849 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 09:27:31.819913 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:31.819857 2584 projected.go:196] Error preparing data for projected volume kube-api-access-t9st6 for pod openshift-network-diagnostics/network-check-target-s6khr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:31.819913 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:31.819906 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6 podName:1d34fb73-3a61-49ce-be61-32b998fa34f1 nodeName:}" failed. No retries permitted until 2026-07-17 09:28:03.819892292 +0000 UTC m=+66.359091249 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9st6" (UniqueName: "kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6") pod "network-check-target-s6khr" (UID: "1d34fb73-3a61-49ce-be61-32b998fa34f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 09:27:32.993246 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:32.993210 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:32.993727 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:32.993210 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:32.993727 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:32.993348 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pkvmq" podUID="ff2f29e6-ee5e-433c-a625-30090f8f5d9c" Jul 17 09:27:32.993727 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:32.993210 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:32.993727 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:32.993453 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vb9ns" podUID="62e2a6ea-beae-40ea-a730-c32f3b172707" Jul 17 09:27:32.993727 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:32.993506 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s6khr" podUID="1d34fb73-3a61-49ce-be61-32b998fa34f1" Jul 17 09:27:33.820860 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.820834 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-236.ec2.internal" event="NodeReady" Jul 17 09:27:33.821054 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.820957 2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 09:27:33.862955 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.862918 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6xm7j"] Jul 17 09:27:33.864561 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.864543 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q545l"] Jul 17 09:27:33.864684 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.864665 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:33.866091 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.866073 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q545l" Jul 17 09:27:33.867190 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.867174 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p8vfs\"" Jul 17 09:27:33.867462 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.867389 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 09:27:33.867529 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.867483 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 09:27:33.868795 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.868776 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 09:27:33.868881 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.868778 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5qcn8\"" Jul 17 09:27:33.868881 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.868865 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 09:27:33.868992 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.868890 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 09:27:33.878165 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.875015 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q545l"] Jul 17 09:27:33.879235 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:33.879214 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6xm7j"] Jul 17 09:27:34.034834 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.034809 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26bf1378-169a-44ee-b4a6-3b8116c5cd52-tmp-dir\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.035284 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.034861 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.035284 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.034961 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbncz\" (UniqueName: \"kubernetes.io/projected/8d9a56e4-3df7-4cda-befd-d758b51cb63c-kube-api-access-rbncz\") pod \"ingress-canary-q545l\" (UID: \"8d9a56e4-3df7-4cda-befd-d758b51cb63c\") " pod="openshift-ingress-canary/ingress-canary-q545l" Jul 17 09:27:34.035284 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.035003 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9a56e4-3df7-4cda-befd-d758b51cb63c-cert\") pod \"ingress-canary-q545l\" (UID: \"8d9a56e4-3df7-4cda-befd-d758b51cb63c\") " pod="openshift-ingress-canary/ingress-canary-q545l" Jul 17 09:27:34.035284 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.035037 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26bf1378-169a-44ee-b4a6-3b8116c5cd52-config-volume\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.035284 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.035072 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6kzp\" (UniqueName: \"kubernetes.io/projected/26bf1378-169a-44ee-b4a6-3b8116c5cd52-kube-api-access-c6kzp\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.136072 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.136041 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbncz\" (UniqueName: \"kubernetes.io/projected/8d9a56e4-3df7-4cda-befd-d758b51cb63c-kube-api-access-rbncz\") pod \"ingress-canary-q545l\" (UID: \"8d9a56e4-3df7-4cda-befd-d758b51cb63c\") " pod="openshift-ingress-canary/ingress-canary-q545l" Jul 17 09:27:34.136200 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.136078 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9a56e4-3df7-4cda-befd-d758b51cb63c-cert\") pod \"ingress-canary-q545l\" (UID: \"8d9a56e4-3df7-4cda-befd-d758b51cb63c\") " pod="openshift-ingress-canary/ingress-canary-q545l" Jul 17 09:27:34.136200 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.136110 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26bf1378-169a-44ee-b4a6-3b8116c5cd52-config-volume\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.136200 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.136142 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6kzp\" (UniqueName: \"kubernetes.io/projected/26bf1378-169a-44ee-b4a6-3b8116c5cd52-kube-api-access-c6kzp\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.136200 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.136160 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26bf1378-169a-44ee-b4a6-3b8116c5cd52-tmp-dir\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.136200 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.136188 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.136469 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:34.136286 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 09:27:34.136469 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:34.136343 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls podName:26bf1378-169a-44ee-b4a6-3b8116c5cd52 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:34.636325789 +0000 UTC m=+37.175524749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls") pod "dns-default-6xm7j" (UID: "26bf1378-169a-44ee-b4a6-3b8116c5cd52") : secret "dns-default-metrics-tls" not found Jul 17 09:27:34.136588 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.136574 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26bf1378-169a-44ee-b4a6-3b8116c5cd52-tmp-dir\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.136778 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.136756 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26bf1378-169a-44ee-b4a6-3b8116c5cd52-config-volume\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.140097 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.140076 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d9a56e4-3df7-4cda-befd-d758b51cb63c-cert\") pod \"ingress-canary-q545l\" (UID: \"8d9a56e4-3df7-4cda-befd-d758b51cb63c\") " pod="openshift-ingress-canary/ingress-canary-q545l" Jul 17 09:27:34.144883 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.144858 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6kzp\" (UniqueName: \"kubernetes.io/projected/26bf1378-169a-44ee-b4a6-3b8116c5cd52-kube-api-access-c6kzp\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.145029 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.145007 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbncz\" (UniqueName: \"kubernetes.io/projected/8d9a56e4-3df7-4cda-befd-d758b51cb63c-kube-api-access-rbncz\") pod \"ingress-canary-q545l\" (UID: \"8d9a56e4-3df7-4cda-befd-d758b51cb63c\") " pod="openshift-ingress-canary/ingress-canary-q545l" Jul 17 09:27:34.185934 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.185890 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q545l" Jul 17 09:27:34.266358 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.266323 2584 generic.go:356] "Generic (PLEG): container finished" podID="ed6be1a8-82df-4378-b517-e3e83bf03857" containerID="88f9e9c9723511603c0bd53faa2a510a51e80a7a6df1389c6905170b37ed3225" exitCode=0 Jul 17 09:27:34.266521 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.266372 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" event={"ID":"ed6be1a8-82df-4378-b517-e3e83bf03857","Type":"ContainerDied","Data":"88f9e9c9723511603c0bd53faa2a510a51e80a7a6df1389c6905170b37ed3225"} Jul 17 09:27:34.318428 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.318385 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q545l"] Jul 17 09:27:34.324387 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:34.324358 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9a56e4_3df7_4cda_befd_d758b51cb63c.slice/crio-8845df7c910f5774aeac9dceb52e895b21781eaaa5123a229ce49c72e993e107 WatchSource:0}: Error finding container 8845df7c910f5774aeac9dceb52e895b21781eaaa5123a229ce49c72e993e107: Status 404 returned error can't find the container with id 8845df7c910f5774aeac9dceb52e895b21781eaaa5123a229ce49c72e993e107 Jul 17 09:27:34.640453 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.640421 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:34.640615 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:34.640562 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 09:27:34.640655 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:34.640621 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls podName:26bf1378-169a-44ee-b4a6-3b8116c5cd52 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:35.640604808 +0000 UTC m=+38.179803761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls") pod "dns-default-6xm7j" (UID: "26bf1378-169a-44ee-b4a6-3b8116c5cd52") : secret "dns-default-metrics-tls" not found Jul 17 09:27:34.877411 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.877361 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt"] Jul 17 09:27:34.879230 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.879189 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:34.880390 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.880363 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kb2fs"] Jul 17 09:27:34.881999 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.881974 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:34.882234 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.882206 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 09:27:34.882482 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.882457 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 17 09:27:34.882652 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.882634 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 09:27:34.885728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.885711 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 17 09:27:34.885845 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.885759 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 09:27:34.885923 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.885902 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wwhtg\"" Jul 17 09:27:34.886025 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.885926 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 17 09:27:34.886025 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.885974 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 09:27:34.886025 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.885980 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-fs5k7\"" Jul 17 09:27:34.886277 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.886260 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 17 09:27:34.893061 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.893006 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 17 09:27:34.893573 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.893553 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt"] Jul 17 09:27:34.899421 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:34.899371 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kb2fs"] Jul 17 09:27:35.006429 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.005489 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg"] Jul 17 09:27:35.010267 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.009918 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4l45k"] Jul 17 09:27:35.010267 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.010019 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:27:35.010267 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.010210 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:27:35.010539 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.010518 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.011123 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.010783 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:27:35.011692 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.011669 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg"] Jul 17 09:27:35.011800 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.011708 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4l45k"] Jul 17 09:27:35.011800 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.011781 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.013040 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.012978 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 09:27:35.013040 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.012993 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 09:27:35.013741 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.013517 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-nvkf6\"" Jul 17 09:27:35.013741 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.013534 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 17 09:27:35.013741 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.013555 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jxtjz\"" Jul 17 09:27:35.013997 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.013862 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 17 09:27:35.014349 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.014181 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 09:27:35.014349 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.014190 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 17 09:27:35.014349 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.014280 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 09:27:35.014550 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.014348 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qg8vm\"" Jul 17 09:27:35.014550 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.014185 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 17 09:27:35.014550 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.014534 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 17 09:27:35.015093 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.014936 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nd9bf\"" Jul 17 09:27:35.015093 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.014991 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 17 09:27:35.015093 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.015053 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 17 09:27:35.015287 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.015151 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 17 09:27:35.020657 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.020636 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 17 09:27:35.044014 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.043987 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.044498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.044020 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c24f15-a691-4a51-a970-a00697042f6e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.044498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.044060 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c24f15-a691-4a51-a970-a00697042f6e-tmp\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.044498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.044129 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.044498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.044162 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9m5\" (UniqueName: \"kubernetes.io/projected/b8c24f15-a691-4a51-a970-a00697042f6e-kube-api-access-hd9m5\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.044498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.044206 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h52p\" (UniqueName: \"kubernetes.io/projected/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-kube-api-access-6h52p\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.044498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.044247 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c24f15-a691-4a51-a970-a00697042f6e-serving-cert\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.044498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.044285 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c24f15-a691-4a51-a970-a00697042f6e-service-ca-bundle\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.044498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.044311 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b8c24f15-a691-4a51-a970-a00697042f6e-snapshots\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.145593 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145500 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c24f15-a691-4a51-a970-a00697042f6e-tmp\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.145593 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145546 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0903913-3b04-448f-b22d-c24a066ad245-config\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.145593 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145575 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr2bt\" (UniqueName: \"kubernetes.io/projected/f0903913-3b04-448f-b22d-c24a066ad245-kube-api-access-dr2bt\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.145818 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145608 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.145818 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145712 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9m5\" (UniqueName: \"kubernetes.io/projected/b8c24f15-a691-4a51-a970-a00697042f6e-kube-api-access-hd9m5\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.145893 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145830 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h52p\" (UniqueName: \"kubernetes.io/projected/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-kube-api-access-6h52p\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.145893 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145867 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-trusted-ca\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.145987 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145915 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c24f15-a691-4a51-a970-a00697042f6e-serving-cert\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.146038 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.145992 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c24f15-a691-4a51-a970-a00697042f6e-tmp\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.146038 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146005 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-serving-cert\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.146140 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146070 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c24f15-a691-4a51-a970-a00697042f6e-service-ca-bundle\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.146140 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146100 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0903913-3b04-448f-b22d-c24a066ad245-serving-cert\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.146241 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146160 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-config\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.146241 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146204 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b8c24f15-a691-4a51-a970-a00697042f6e-snapshots\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.146342 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146246 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.146342 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146276 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c24f15-a691-4a51-a970-a00697042f6e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.146342 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146303 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvc7\" (UniqueName: \"kubernetes.io/projected/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-kube-api-access-pzvc7\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.146527 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146392 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.146527 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:35.146443 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:35.146527 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:35.146502 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls podName:1ed006d9-ebc3-4e86-b54a-4e28aa293ba9 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:35.646487534 +0000 UTC m=+38.185686478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-svxdt" (UID: "1ed006d9-ebc3-4e86-b54a-4e28aa293ba9") : secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:35.146680 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146649 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c24f15-a691-4a51-a970-a00697042f6e-service-ca-bundle\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.146776 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.146756 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b8c24f15-a691-4a51-a970-a00697042f6e-snapshots\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.147141 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.147120 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c24f15-a691-4a51-a970-a00697042f6e-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.149674 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.149653 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c24f15-a691-4a51-a970-a00697042f6e-serving-cert\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.154306 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.154287 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9m5\" (UniqueName: \"kubernetes.io/projected/b8c24f15-a691-4a51-a970-a00697042f6e-kube-api-access-hd9m5\") pod \"insights-operator-555d9d7885-kb2fs\" (UID: \"b8c24f15-a691-4a51-a970-a00697042f6e\") " pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.154866 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.154824 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h52p\" (UniqueName: \"kubernetes.io/projected/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-kube-api-access-6h52p\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.208531 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.208500 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-kb2fs" Jul 17 09:27:35.247419 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.246775 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-trusted-ca\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.247419 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.246891 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-serving-cert\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.247419 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.246922 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0903913-3b04-448f-b22d-c24a066ad245-serving-cert\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.247419 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.246954 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-config\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.247419 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.247015 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvc7\" (UniqueName: \"kubernetes.io/projected/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-kube-api-access-pzvc7\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.247419 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.247066 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0903913-3b04-448f-b22d-c24a066ad245-config\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.247419 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.247093 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr2bt\" (UniqueName: \"kubernetes.io/projected/f0903913-3b04-448f-b22d-c24a066ad245-kube-api-access-dr2bt\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.247867 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.247649 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-trusted-ca\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.248242 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.248053 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-config\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.248242 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.248120 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0903913-3b04-448f-b22d-c24a066ad245-config\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.250797 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.250697 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0903913-3b04-448f-b22d-c24a066ad245-serving-cert\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.250797 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.250724 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-serving-cert\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.256365 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.256333 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr2bt\" (UniqueName: \"kubernetes.io/projected/f0903913-3b04-448f-b22d-c24a066ad245-kube-api-access-dr2bt\") pod \"service-ca-operator-5bcbc94c78-frxhg\" (UID: \"f0903913-3b04-448f-b22d-c24a066ad245\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.258202 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.258181 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvc7\" (UniqueName: \"kubernetes.io/projected/8de78fc2-54b3-498b-bd7f-4d38b3a23bc7-kube-api-access-pzvc7\") pod \"console-operator-565b7b569d-4l45k\" (UID: \"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7\") " pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.272823 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.272789 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q545l" event={"ID":"8d9a56e4-3df7-4cda-befd-d758b51cb63c","Type":"ContainerStarted","Data":"8845df7c910f5774aeac9dceb52e895b21781eaaa5123a229ce49c72e993e107"} Jul 17 09:27:35.277062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.277034 2584 generic.go:356] "Generic (PLEG): container finished" podID="ed6be1a8-82df-4378-b517-e3e83bf03857" containerID="eb254081eca385bb0d9007237fa3169293225f6a8b9c6ade84d70ed3a55570fe" exitCode=0 Jul 17 09:27:35.277148 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.277088 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" event={"ID":"ed6be1a8-82df-4378-b517-e3e83bf03857","Type":"ContainerDied","Data":"eb254081eca385bb0d9007237fa3169293225f6a8b9c6ade84d70ed3a55570fe"} Jul 17 09:27:35.338446 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.338217 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" Jul 17 09:27:35.351922 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.351895 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:35.359882 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.359818 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-kb2fs"] Jul 17 09:27:35.372130 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:35.370980 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c24f15_a691_4a51_a970_a00697042f6e.slice/crio-44e4299ed52c13316d1742d55a20da838a428391780e5ac92f874edb72090d44 WatchSource:0}: Error finding container 44e4299ed52c13316d1742d55a20da838a428391780e5ac92f874edb72090d44: Status 404 returned error can't find the container with id 44e4299ed52c13316d1742d55a20da838a428391780e5ac92f874edb72090d44 Jul 17 09:27:35.500273 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.500235 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg"] Jul 17 09:27:35.503685 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:35.503649 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0903913_3b04_448f_b22d_c24a066ad245.slice/crio-98ccbc05c3fc7d525075624f4fd2b04bf5f338653fb4b25d65ecef0da30810a0 WatchSource:0}: Error finding container 98ccbc05c3fc7d525075624f4fd2b04bf5f338653fb4b25d65ecef0da30810a0: Status 404 returned error can't find the container with id 98ccbc05c3fc7d525075624f4fd2b04bf5f338653fb4b25d65ecef0da30810a0 Jul 17 09:27:35.514751 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.514721 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-4l45k"] Jul 17 09:27:35.519379 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:35.519355 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de78fc2_54b3_498b_bd7f_4d38b3a23bc7.slice/crio-b7ae73370f1dae1b310b9ec1f309ac2cbe4e259da77ba2d66ac613f8178cedfa WatchSource:0}: Error finding container b7ae73370f1dae1b310b9ec1f309ac2cbe4e259da77ba2d66ac613f8178cedfa: Status 404 returned error can't find the container with id b7ae73370f1dae1b310b9ec1f309ac2cbe4e259da77ba2d66ac613f8178cedfa Jul 17 09:27:35.650597 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.650559 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:35.650747 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:35.650627 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:35.650747 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:35.650735 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:35.650814 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:35.650741 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 09:27:35.650846 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:35.650820 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls podName:26bf1378-169a-44ee-b4a6-3b8116c5cd52 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:37.65079985 +0000 UTC m=+40.189998812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls") pod "dns-default-6xm7j" (UID: "26bf1378-169a-44ee-b4a6-3b8116c5cd52") : secret "dns-default-metrics-tls" not found Jul 17 09:27:35.650846 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:35.650841 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls podName:1ed006d9-ebc3-4e86-b54a-4e28aa293ba9 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:36.650831013 +0000 UTC m=+39.190029960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-svxdt" (UID: "1ed006d9-ebc3-4e86-b54a-4e28aa293ba9") : secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:36.281282 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:36.281225 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" event={"ID":"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7","Type":"ContainerStarted","Data":"b7ae73370f1dae1b310b9ec1f309ac2cbe4e259da77ba2d66ac613f8178cedfa"} Jul 17 09:27:36.284874 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:36.284852 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" event={"ID":"ed6be1a8-82df-4378-b517-e3e83bf03857","Type":"ContainerStarted","Data":"93ec7dcfcf12d89578080b563faeaf7a00fa9a22b6e37c8d3324db2c404f0108"} Jul 17 09:27:36.286332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:36.286312 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" event={"ID":"f0903913-3b04-448f-b22d-c24a066ad245","Type":"ContainerStarted","Data":"98ccbc05c3fc7d525075624f4fd2b04bf5f338653fb4b25d65ecef0da30810a0"} Jul 17 09:27:36.287572 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:36.287548 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kb2fs" event={"ID":"b8c24f15-a691-4a51-a970-a00697042f6e","Type":"ContainerStarted","Data":"44e4299ed52c13316d1742d55a20da838a428391780e5ac92f874edb72090d44"} Jul 17 09:27:36.308602 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:36.308540 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8mqkt" podStartSLOduration=8.09366699 podStartE2EDuration="38.308522241s" totalImagesPullingTime="30.214855251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:36.308255556 +0000 UTC m=+38.847454524" watchObservedRunningTime="2026-07-17 09:27:36.308522241 +0000 UTC m=+38.847721206" Jul 17 09:27:36.659797 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:36.659765 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:36.659961 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:36.659900 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:36.659961 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:36.659958 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls podName:1ed006d9-ebc3-4e86-b54a-4e28aa293ba9 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:38.659944971 +0000 UTC m=+41.199143920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-svxdt" (UID: "1ed006d9-ebc3-4e86-b54a-4e28aa293ba9") : secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:37.293174 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:37.292763 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q545l" event={"ID":"8d9a56e4-3df7-4cda-befd-d758b51cb63c","Type":"ContainerStarted","Data":"762c2a2f22f42c7e3fffa33ed7429fc9f53c8ed9db43640964bca822f85d2b3d"} Jul 17 09:27:37.308714 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:37.308453 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q545l" podStartSLOduration=2.080405164 podStartE2EDuration="4.308431813s" totalImagesPullingTime="2.228026649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:37.307839529 +0000 UTC m=+39.847038497" watchObservedRunningTime="2026-07-17 09:27:37.308431813 +0000 UTC m=+39.847630781" Jul 17 09:27:37.669533 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:37.669477 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:37.669734 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:37.669666 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 09:27:37.669782 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:37.669759 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls podName:26bf1378-169a-44ee-b4a6-3b8116c5cd52 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:41.669737011 +0000 UTC m=+44.208935958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls") pod "dns-default-6xm7j" (UID: "26bf1378-169a-44ee-b4a6-3b8116c5cd52") : secret "dns-default-metrics-tls" not found Jul 17 09:27:38.680910 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:38.678838 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:38.680910 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:38.679521 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:38.680910 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:38.679626 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls podName:1ed006d9-ebc3-4e86-b54a-4e28aa293ba9 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:42.679604641 +0000 UTC m=+45.218803601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-svxdt" (UID: "1ed006d9-ebc3-4e86-b54a-4e28aa293ba9") : secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:38.957147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:38.957061 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c"] Jul 17 09:27:38.977357 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:38.977329 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c"] Jul 17 09:27:38.977496 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:38.977453 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" Jul 17 09:27:38.979901 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:38.979869 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 17 09:27:38.980023 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:38.979962 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 17 09:27:38.980972 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:38.980950 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vswxk\"" Jul 17 09:27:39.081523 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:39.081489 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtp8\" (UniqueName: \"kubernetes.io/projected/7b905e46-4bf8-45ac-9704-72dd44ba03cc-kube-api-access-vvtp8\") pod \"migrator-7cc9c7ffd7-zld8c\" (UID: \"7b905e46-4bf8-45ac-9704-72dd44ba03cc\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" Jul 17 09:27:39.182024 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:39.181983 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtp8\" (UniqueName: \"kubernetes.io/projected/7b905e46-4bf8-45ac-9704-72dd44ba03cc-kube-api-access-vvtp8\") pod \"migrator-7cc9c7ffd7-zld8c\" (UID: \"7b905e46-4bf8-45ac-9704-72dd44ba03cc\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" Jul 17 09:27:39.192030 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:39.191997 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtp8\" (UniqueName: \"kubernetes.io/projected/7b905e46-4bf8-45ac-9704-72dd44ba03cc-kube-api-access-vvtp8\") pod \"migrator-7cc9c7ffd7-zld8c\" (UID: \"7b905e46-4bf8-45ac-9704-72dd44ba03cc\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" Jul 17 09:27:39.286847 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:39.286758 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" Jul 17 09:27:40.179003 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.178933 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c"] Jul 17 09:27:40.183309 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:27:40.183276 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b905e46_4bf8_45ac_9704_72dd44ba03cc.slice/crio-70e0989f4a567068e0af85be0d7271f4ed8f50d94b40eb0c468b08c1dc363d32 WatchSource:0}: Error finding container 70e0989f4a567068e0af85be0d7271f4ed8f50d94b40eb0c468b08c1dc363d32: Status 404 returned error can't find the container with id 70e0989f4a567068e0af85be0d7271f4ed8f50d94b40eb0c468b08c1dc363d32 Jul 17 09:27:40.304338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.304309 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" event={"ID":"7b905e46-4bf8-45ac-9704-72dd44ba03cc","Type":"ContainerStarted","Data":"70e0989f4a567068e0af85be0d7271f4ed8f50d94b40eb0c468b08c1dc363d32"} Jul 17 09:27:40.305972 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.305951 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/0.log" Jul 17 09:27:40.306079 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.305993 2584 generic.go:356] "Generic (PLEG): container finished" podID="8de78fc2-54b3-498b-bd7f-4d38b3a23bc7" containerID="eeaa45023ee206d8ed6f067aa8aba60903ba4e47ea7df3bbf87a28835706dc90" exitCode=255 Jul 17 09:27:40.306136 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.306074 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" event={"ID":"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7","Type":"ContainerDied","Data":"eeaa45023ee206d8ed6f067aa8aba60903ba4e47ea7df3bbf87a28835706dc90"} Jul 17 09:27:40.306342 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.306325 2584 scope.go:117] "RemoveContainer" containerID="eeaa45023ee206d8ed6f067aa8aba60903ba4e47ea7df3bbf87a28835706dc90" Jul 17 09:27:40.307628 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.307607 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" event={"ID":"f0903913-3b04-448f-b22d-c24a066ad245","Type":"ContainerStarted","Data":"f8948fa708d1dd996ade6043fd7f03a88b9c8fe35c38b0d131fc717cc2019b86"} Jul 17 09:27:40.309145 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.309122 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kb2fs" event={"ID":"b8c24f15-a691-4a51-a970-a00697042f6e","Type":"ContainerStarted","Data":"8d6fc0ce569824578856be053337bf600f3a0c684d62fc782778679a3c965e6d"} Jul 17 09:27:40.335813 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.335769 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" podStartSLOduration=1.793000335 podStartE2EDuration="6.33575494s" totalImagesPullingTime="4.542754605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:40.334880906 +0000 UTC m=+42.874079873" watchObservedRunningTime="2026-07-17 09:27:40.33575494 +0000 UTC m=+42.874953905" Jul 17 09:27:40.351336 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:40.351239 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-kb2fs" podStartSLOduration=1.678091542 podStartE2EDuration="6.351221568s" totalImagesPullingTime="4.673130026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:40.349408811 +0000 UTC m=+42.888607772" watchObservedRunningTime="2026-07-17 09:27:40.351221568 +0000 UTC m=+42.890420535" Jul 17 09:27:41.109989 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.104997 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-477zg"] Jul 17 09:27:41.127700 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.127640 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-477zg"] Jul 17 09:27:41.127899 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.127787 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.130153 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.130112 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 09:27:41.130312 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.130168 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gmxxn\"" Jul 17 09:27:41.130312 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.130294 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 09:27:41.197886 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.197840 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e3ef95a-3fc4-4565-8c77-df73139c624f-data-volume\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.198275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.197932 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pk4g\" (UniqueName: \"kubernetes.io/projected/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-api-access-6pk4g\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.198275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.198014 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.198275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.198039 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e3ef95a-3fc4-4565-8c77-df73139c624f-crio-socket\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.198275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.198072 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.298587 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.298544 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.298744 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.298605 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e3ef95a-3fc4-4565-8c77-df73139c624f-data-volume\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.298744 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.298640 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pk4g\" (UniqueName: \"kubernetes.io/projected/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-api-access-6pk4g\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.298744 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.298693 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 09:27:41.298880 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.298746 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 09:27:41.298880 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.298761 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:41.798743364 +0000 UTC m=+44.337942308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : secret "insights-runtime-extractor-tls" not found Jul 17 09:27:41.298880 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.298699 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.298880 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.298806 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:41.798791396 +0000 UTC m=+44.337990357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : configmap "kube-rbac-proxy" not found Jul 17 09:27:41.298880 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.298847 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e3ef95a-3fc4-4565-8c77-df73139c624f-crio-socket\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.299165 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.299012 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e3ef95a-3fc4-4565-8c77-df73139c624f-data-volume\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.299165 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.299066 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e3ef95a-3fc4-4565-8c77-df73139c624f-crio-socket\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.306529 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.306508 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pk4g\" (UniqueName: \"kubernetes.io/projected/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-api-access-6pk4g\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.313845 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.313826 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:27:41.314230 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.314216 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/0.log" Jul 17 09:27:41.314289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.314252 2584 generic.go:356] "Generic (PLEG): container finished" podID="8de78fc2-54b3-498b-bd7f-4d38b3a23bc7" containerID="75637e2c3b63e6c6b6de6a0e2732909caf108509e9fb1d3e559928f48a7b57c1" exitCode=255 Jul 17 09:27:41.314338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.314318 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" event={"ID":"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7","Type":"ContainerDied","Data":"75637e2c3b63e6c6b6de6a0e2732909caf108509e9fb1d3e559928f48a7b57c1"} Jul 17 09:27:41.314387 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.314376 2584 scope.go:117] "RemoveContainer" containerID="eeaa45023ee206d8ed6f067aa8aba60903ba4e47ea7df3bbf87a28835706dc90" Jul 17 09:27:41.314695 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.314675 2584 scope.go:117] "RemoveContainer" containerID="75637e2c3b63e6c6b6de6a0e2732909caf108509e9fb1d3e559928f48a7b57c1" Jul 17 09:27:41.314856 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.314833 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4l45k_openshift-console-operator(8de78fc2-54b3-498b-bd7f-4d38b3a23bc7)\"" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" podUID="8de78fc2-54b3-498b-bd7f-4d38b3a23bc7" Jul 17 09:27:41.703447 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.703391 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:41.703589 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.703543 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 09:27:41.703646 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.703607 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls podName:26bf1378-169a-44ee-b4a6-3b8116c5cd52 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:49.703592189 +0000 UTC m=+52.242791133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls") pod "dns-default-6xm7j" (UID: "26bf1378-169a-44ee-b4a6-3b8116c5cd52") : secret "dns-default-metrics-tls" not found Jul 17 09:27:41.804647 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.804611 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.804822 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:41.804662 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:41.804822 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.804758 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 09:27:41.804926 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.804833 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:42.804811576 +0000 UTC m=+45.344010524 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : configmap "kube-rbac-proxy" not found Jul 17 09:27:41.804926 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.804880 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 09:27:41.804926 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:41.804923 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:42.804909426 +0000 UTC m=+45.344108369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : secret "insights-runtime-extractor-tls" not found Jul 17 09:27:42.318963 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:42.318894 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:27:42.319385 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:42.319246 2584 scope.go:117] "RemoveContainer" containerID="75637e2c3b63e6c6b6de6a0e2732909caf108509e9fb1d3e559928f48a7b57c1" Jul 17 09:27:42.319454 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:42.319425 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4l45k_openshift-console-operator(8de78fc2-54b3-498b-bd7f-4d38b3a23bc7)\"" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" podUID="8de78fc2-54b3-498b-bd7f-4d38b3a23bc7" Jul 17 09:27:42.320741 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:42.320717 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" event={"ID":"7b905e46-4bf8-45ac-9704-72dd44ba03cc","Type":"ContainerStarted","Data":"1670027e6e2039174b2070de4a3e1f133d92f083946d341918137d02e1c4eac6"} Jul 17 09:27:42.320847 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:42.320763 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" event={"ID":"7b905e46-4bf8-45ac-9704-72dd44ba03cc","Type":"ContainerStarted","Data":"1d544f5a03480b4cb4cda3bfcdd4917739a1db4dc9a00fc730e85e972a8af568"} Jul 17 09:27:42.349065 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:42.349012 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-zld8c" podStartSLOduration=2.510402541 podStartE2EDuration="4.348996596s" totalImagesPullingTime="1.838594055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:42.348880495 +0000 UTC m=+44.888079464" watchObservedRunningTime="2026-07-17 09:27:42.348996596 +0000 UTC m=+44.888195558" Jul 17 09:27:42.713033 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:42.712998 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:42.713220 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:42.713170 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:42.713265 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:42.713232 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls podName:1ed006d9-ebc3-4e86-b54a-4e28aa293ba9 nodeName:}" failed. No retries permitted until 2026-07-17 09:27:50.713216566 +0000 UTC m=+53.252415513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-svxdt" (UID: "1ed006d9-ebc3-4e86-b54a-4e28aa293ba9") : secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:42.814392 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:42.814352 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:42.814584 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:42.814438 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:42.814584 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:42.814497 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 09:27:42.814584 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:42.814565 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 09:27:42.814584 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:42.814570 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:44.814549604 +0000 UTC m=+47.353748570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : configmap "kube-rbac-proxy" not found Jul 17 09:27:42.814718 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:42.814601 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:44.814589862 +0000 UTC m=+47.353788817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : secret "insights-runtime-extractor-tls" not found Jul 17 09:27:44.035271 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:44.035241 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zvws7_eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7/dns-node-resolver/0.log" Jul 17 09:27:44.829407 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:44.829368 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:44.829588 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:44.829500 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:44.829588 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:44.829520 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 09:27:44.829588 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:44.829576 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 09:27:44.829588 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:44.829588 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:48.829572254 +0000 UTC m=+51.368771197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : secret "insights-runtime-extractor-tls" not found Jul 17 09:27:44.829727 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:44.829616 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:48.829604998 +0000 UTC m=+51.368803943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : configmap "kube-rbac-proxy" not found Jul 17 09:27:44.835092 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:44.835072 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hkg55_5ab356ac-a51c-44dc-b850-1a0f814a0b0f/node-ca/0.log" Jul 17 09:27:45.352661 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:45.352626 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:45.353028 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:45.352678 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:45.353028 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:45.352998 2584 scope.go:117] "RemoveContainer" containerID="75637e2c3b63e6c6b6de6a0e2732909caf108509e9fb1d3e559928f48a7b57c1" Jul 17 09:27:45.353159 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:45.353134 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-4l45k_openshift-console-operator(8de78fc2-54b3-498b-bd7f-4d38b3a23bc7)\"" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" podUID="8de78fc2-54b3-498b-bd7f-4d38b3a23bc7" Jul 17 09:27:45.436137 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:45.436110 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q545l_8d9a56e4-3df7-4cda-befd-d758b51cb63c/serve-healthcheck-canary/0.log" Jul 17 09:27:45.837075 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:45.837002 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-zld8c_7b905e46-4bf8-45ac-9704-72dd44ba03cc/migrator/0.log" Jul 17 09:27:46.036081 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:46.036051 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-zld8c_7b905e46-4bf8-45ac-9704-72dd44ba03cc/graceful-termination/0.log" Jul 17 09:27:48.864670 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:48.864631 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:48.865140 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:48.864684 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:48.865140 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:48.864754 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 09:27:48.865140 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:48.864769 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 09:27:48.865140 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:48.864818 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:56.864804601 +0000 UTC m=+59.404003545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : secret "insights-runtime-extractor-tls" not found Jul 17 09:27:48.865140 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:48.864856 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:27:56.864834908 +0000 UTC m=+59.404033855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : configmap "kube-rbac-proxy" not found Jul 17 09:27:49.771773 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:49.771737 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:27:49.771963 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:49.771878 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 09:27:49.771963 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:49.771942 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls podName:26bf1378-169a-44ee-b4a6-3b8116c5cd52 nodeName:}" failed. No retries permitted until 2026-07-17 09:28:05.771924347 +0000 UTC m=+68.311123307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls") pod "dns-default-6xm7j" (UID: "26bf1378-169a-44ee-b4a6-3b8116c5cd52") : secret "dns-default-metrics-tls" not found Jul 17 09:27:50.780575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:50.780525 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:27:50.781028 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:50.780675 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:50.781028 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:50.780743 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls podName:1ed006d9-ebc3-4e86-b54a-4e28aa293ba9 nodeName:}" failed. No retries permitted until 2026-07-17 09:28:06.780727175 +0000 UTC m=+69.319926118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-svxdt" (UID: "1ed006d9-ebc3-4e86-b54a-4e28aa293ba9") : secret "cluster-monitoring-operator-tls" not found Jul 17 09:27:56.933461 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:56.933413 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:56.933939 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:56.933476 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:56.933939 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:56.933587 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 09:27:56.933939 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:27:56.933646 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls podName:7e3ef95a-3fc4-4565-8c77-df73139c624f nodeName:}" failed. No retries permitted until 2026-07-17 09:28:12.933632514 +0000 UTC m=+75.472831458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-477zg" (UID: "7e3ef95a-3fc4-4565-8c77-df73139c624f") : secret "insights-runtime-extractor-tls" not found Jul 17 09:27:56.933939 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:56.933895 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e3ef95a-3fc4-4565-8c77-df73139c624f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:27:57.254739 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:57.254660 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bwcc" Jul 17 09:27:58.993748 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:58.993716 2584 scope.go:117] "RemoveContainer" containerID="75637e2c3b63e6c6b6de6a0e2732909caf108509e9fb1d3e559928f48a7b57c1" Jul 17 09:27:59.372450 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:59.372426 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:27:59.372704 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:59.372508 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" event={"ID":"8de78fc2-54b3-498b-bd7f-4d38b3a23bc7","Type":"ContainerStarted","Data":"b62cbd148d58ac9c63d567ae3ceef90f2fd83182f11468d2b8acb7713326a95e"} Jul 17 09:27:59.372770 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:59.372748 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:27:59.395671 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:59.395618 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" podStartSLOduration=20.862212775 podStartE2EDuration="25.395599324s" totalImagesPullingTime="4.533386549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:27:59.394701599 +0000 UTC m=+61.933900562" watchObservedRunningTime="2026-07-17 09:27:59.395599324 +0000 UTC m=+61.934798289" Jul 17 09:27:59.777864 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:27:59.777772 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-4l45k" Jul 17 09:28:03.785375 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.785324 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:28:03.785375 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.785379 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:28:03.788498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.788480 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 09:28:03.788547 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.788481 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 09:28:03.797961 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.797936 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ff2f29e6-ee5e-433c-a625-30090f8f5d9c-original-pull-secret\") pod \"global-pull-secret-syncer-pkvmq\" (UID: \"ff2f29e6-ee5e-433c-a625-30090f8f5d9c\") " pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:28:03.798089 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.797972 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2a6ea-beae-40ea-a730-c32f3b172707-metrics-certs\") pod \"network-metrics-daemon-vb9ns\" (UID: \"62e2a6ea-beae-40ea-a730-c32f3b172707\") " pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:28:03.824244 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.824195 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pkvmq" Jul 17 09:28:03.833658 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.833638 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jxtjz\"" Jul 17 09:28:03.842016 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.841992 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vb9ns" Jul 17 09:28:03.886665 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.886465 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:28:03.889763 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.889737 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 09:28:03.900432 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.900245 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 09:28:03.912107 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.912033 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9st6\" (UniqueName: \"kubernetes.io/projected/1d34fb73-3a61-49ce-be61-32b998fa34f1-kube-api-access-t9st6\") pod \"network-check-target-s6khr\" (UID: \"1d34fb73-3a61-49ce-be61-32b998fa34f1\") " pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:28:03.980819 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.980786 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pkvmq"] Jul 17 09:28:03.984147 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:03.984115 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2f29e6_ee5e_433c_a625_30090f8f5d9c.slice/crio-206a851cb59c7ec6d8e50ff54382e6d28587bfbb92ffca39a370455dab22910d WatchSource:0}: Error finding container 206a851cb59c7ec6d8e50ff54382e6d28587bfbb92ffca39a370455dab22910d: Status 404 returned error can't find the container with id 206a851cb59c7ec6d8e50ff54382e6d28587bfbb92ffca39a370455dab22910d Jul 17 09:28:03.987608 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:03.987586 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vb9ns"] Jul 17 09:28:03.992259 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:03.992234 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e2a6ea_beae_40ea_a730_c32f3b172707.slice/crio-cbc93937dda1509494111d4c783faee56b5939f313c37d6c836f8b445bd91cdf WatchSource:0}: Error finding container cbc93937dda1509494111d4c783faee56b5939f313c37d6c836f8b445bd91cdf: Status 404 returned error can't find the container with id cbc93937dda1509494111d4c783faee56b5939f313c37d6c836f8b445bd91cdf Jul 17 09:28:04.147897 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.147872 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qg8vm\"" Jul 17 09:28:04.155941 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.155918 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:28:04.276699 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.276663 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s6khr"] Jul 17 09:28:04.279883 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:04.279859 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d34fb73_3a61_49ce_be61_32b998fa34f1.slice/crio-1929035dd7eb427a5738845b2d157a6632f1145c3cf904f365431ec761006361 WatchSource:0}: Error finding container 1929035dd7eb427a5738845b2d157a6632f1145c3cf904f365431ec761006361: Status 404 returned error can't find the container with id 1929035dd7eb427a5738845b2d157a6632f1145c3cf904f365431ec761006361 Jul 17 09:28:04.388626 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.388591 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s6khr" event={"ID":"1d34fb73-3a61-49ce-be61-32b998fa34f1","Type":"ContainerStarted","Data":"1929035dd7eb427a5738845b2d157a6632f1145c3cf904f365431ec761006361"} Jul 17 09:28:04.390116 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.390084 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb9ns" event={"ID":"62e2a6ea-beae-40ea-a730-c32f3b172707","Type":"ContainerStarted","Data":"cbc93937dda1509494111d4c783faee56b5939f313c37d6c836f8b445bd91cdf"} Jul 17 09:28:04.391637 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.391610 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pkvmq" event={"ID":"ff2f29e6-ee5e-433c-a625-30090f8f5d9c","Type":"ContainerStarted","Data":"206a851cb59c7ec6d8e50ff54382e6d28587bfbb92ffca39a370455dab22910d"} Jul 17 09:28:04.597238 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.597200 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j"] Jul 17 09:28:04.600327 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.599909 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.604326 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.604147 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 17 09:28:04.604326 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.604205 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 17 09:28:04.604326 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.604295 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 17 09:28:04.604568 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.604416 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 17 09:28:04.608829 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.608797 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76898c686b-x7dz5"] Jul 17 09:28:04.611018 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.610995 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.613536 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.613516 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 09:28:04.613790 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.613771 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-45kr4\"" Jul 17 09:28:04.616372 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.616169 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 09:28:04.616491 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.616382 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 09:28:04.625261 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.625241 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j"] Jul 17 09:28:04.625843 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.625818 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 09:28:04.642120 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.642087 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76898c686b-x7dz5"] Jul 17 09:28:04.691713 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.691681 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-klusterlet-config\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.691886 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.691733 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-tls\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.691886 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.691775 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-image-registry-private-configuration\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.691886 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.691821 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-trusted-ca\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.691886 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.691867 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-certificates\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.691886 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.691887 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gr88\" (UniqueName: \"kubernetes.io/projected/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-kube-api-access-6gr88\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.692277 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.691906 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-installation-pull-secrets\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.692277 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.691935 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv78f\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-kube-api-access-zv78f\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.692277 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.692000 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-tmp\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.692277 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.692196 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-bound-sa-token\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.692277 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.692232 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc51f866-4e1b-4026-976e-bcd8382800a0-ca-trust-extracted\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.737222 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.736713 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76898c686b-x7dz5"] Jul 17 09:28:04.737222 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:28:04.737015 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-zv78f registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-76898c686b-x7dz5" podUID="dc51f866-4e1b-4026-976e-bcd8382800a0" Jul 17 09:28:04.791525 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.790473 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7675b5767f-wp4r5"] Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.792631 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.792851 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-klusterlet-config\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.792892 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-tls\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.792931 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-image-registry-private-configuration\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.792974 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-trusted-ca\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793006 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-certificates\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793029 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gr88\" (UniqueName: \"kubernetes.io/projected/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-kube-api-access-6gr88\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793055 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-installation-pull-secrets\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793095 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv78f\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-kube-api-access-zv78f\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793131 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-tmp\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793376 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-bound-sa-token\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793430 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc51f866-4e1b-4026-976e-bcd8382800a0-ca-trust-extracted\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793526 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-tmp\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.794228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.793754 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc51f866-4e1b-4026-976e-bcd8382800a0-ca-trust-extracted\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.795011 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.794940 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-certificates\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.795235 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.795207 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-trusted-ca\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.797343 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.797295 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-tls\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.798182 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.798135 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-image-registry-private-configuration\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.798914 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.798888 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-installation-pull-secrets\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.800244 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.800222 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-klusterlet-config\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.817715 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.814432 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-bound-sa-token\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.817715 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.816548 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7675b5767f-wp4r5"] Jul 17 09:28:04.819781 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.819754 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv78f\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-kube-api-access-zv78f\") pod \"image-registry-76898c686b-x7dz5\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:04.821318 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.821274 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gr88\" (UniqueName: \"kubernetes.io/projected/7c46b1c8-d9ec-475a-9ce7-37264a5b467c-kube-api-access-6gr88\") pod \"klusterlet-addon-workmgr-658858dd96-7z45j\" (UID: \"7c46b1c8-d9ec-475a-9ce7-37264a5b467c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.894960 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.894842 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ce68b72-471e-40be-a4cf-0c493ccfcca7-registry-certificates\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.894960 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.894909 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ce68b72-471e-40be-a4cf-0c493ccfcca7-trusted-ca\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.895195 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.894989 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ce68b72-471e-40be-a4cf-0c493ccfcca7-image-registry-private-configuration\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.895195 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.895017 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-registry-tls\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.895195 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.895039 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-bound-sa-token\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.895195 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.895067 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mppr\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-kube-api-access-5mppr\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.895195 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.895097 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ce68b72-471e-40be-a4cf-0c493ccfcca7-ca-trust-extracted\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.895195 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.895118 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ce68b72-471e-40be-a4cf-0c493ccfcca7-installation-pull-secrets\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.912668 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.912632 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:04.996979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.995601 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ce68b72-471e-40be-a4cf-0c493ccfcca7-image-registry-private-configuration\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.996979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.995653 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-registry-tls\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.996979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.995681 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-bound-sa-token\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.996979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.995710 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mppr\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-kube-api-access-5mppr\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.996979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.995743 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ce68b72-471e-40be-a4cf-0c493ccfcca7-ca-trust-extracted\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.996979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.995771 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ce68b72-471e-40be-a4cf-0c493ccfcca7-installation-pull-secrets\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.996979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.995845 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ce68b72-471e-40be-a4cf-0c493ccfcca7-registry-certificates\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.996979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.995870 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ce68b72-471e-40be-a4cf-0c493ccfcca7-trusted-ca\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.997544 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.997141 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9ce68b72-471e-40be-a4cf-0c493ccfcca7-trusted-ca\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.997544 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.997483 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9ce68b72-471e-40be-a4cf-0c493ccfcca7-ca-trust-extracted\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.998035 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.998010 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9ce68b72-471e-40be-a4cf-0c493ccfcca7-registry-certificates\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.999669 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.999620 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9ce68b72-471e-40be-a4cf-0c493ccfcca7-image-registry-private-configuration\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:04.999669 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:04.999654 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-registry-tls\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:05.001231 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.001199 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9ce68b72-471e-40be-a4cf-0c493ccfcca7-installation-pull-secrets\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:05.018957 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.018931 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-bound-sa-token\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:05.019223 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.019198 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mppr\" (UniqueName: \"kubernetes.io/projected/9ce68b72-471e-40be-a4cf-0c493ccfcca7-kube-api-access-5mppr\") pod \"image-registry-7675b5767f-wp4r5\" (UID: \"9ce68b72-471e-40be-a4cf-0c493ccfcca7\") " pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:05.085418 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.085370 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j"] Jul 17 09:28:05.141705 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.141449 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:05.254104 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:05.254050 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c46b1c8_d9ec_475a_9ce7_37264a5b467c.slice/crio-4a389ac29caea856bebe1990986cbe2b96af9c162091bad0bbb120691e9f69af WatchSource:0}: Error finding container 4a389ac29caea856bebe1990986cbe2b96af9c162091bad0bbb120691e9f69af: Status 404 returned error can't find the container with id 4a389ac29caea856bebe1990986cbe2b96af9c162091bad0bbb120691e9f69af Jul 17 09:28:05.402090 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.402052 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:05.402090 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.402078 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" event={"ID":"7c46b1c8-d9ec-475a-9ce7-37264a5b467c","Type":"ContainerStarted","Data":"4a389ac29caea856bebe1990986cbe2b96af9c162091bad0bbb120691e9f69af"} Jul 17 09:28:05.418373 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.418212 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7675b5767f-wp4r5"] Jul 17 09:28:05.427949 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:05.427902 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce68b72_471e_40be_a4cf_0c493ccfcca7.slice/crio-df6c34713fa5543580335300baf854e3042f91a9c8003ea59935382f46775444 WatchSource:0}: Error finding container df6c34713fa5543580335300baf854e3042f91a9c8003ea59935382f46775444: Status 404 returned error can't find the container with id df6c34713fa5543580335300baf854e3042f91a9c8003ea59935382f46775444 Jul 17 09:28:05.433104 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.433075 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:05.500481 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500454 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-trusted-ca\") pod \"dc51f866-4e1b-4026-976e-bcd8382800a0\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " Jul 17 09:28:05.500578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500488 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-bound-sa-token\") pod \"dc51f866-4e1b-4026-976e-bcd8382800a0\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " Jul 17 09:28:05.500578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500517 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv78f\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-kube-api-access-zv78f\") pod \"dc51f866-4e1b-4026-976e-bcd8382800a0\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " Jul 17 09:28:05.500578 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500566 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-tls\") pod \"dc51f866-4e1b-4026-976e-bcd8382800a0\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " Jul 17 09:28:05.500674 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500591 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-certificates\") pod \"dc51f866-4e1b-4026-976e-bcd8382800a0\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " Jul 17 09:28:05.500674 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500619 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-installation-pull-secrets\") pod \"dc51f866-4e1b-4026-976e-bcd8382800a0\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " Jul 17 09:28:05.500674 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500678 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-image-registry-private-configuration\") pod \"dc51f866-4e1b-4026-976e-bcd8382800a0\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " Jul 17 09:28:05.500770 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500705 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc51f866-4e1b-4026-976e-bcd8382800a0-ca-trust-extracted\") pod \"dc51f866-4e1b-4026-976e-bcd8382800a0\" (UID: \"dc51f866-4e1b-4026-976e-bcd8382800a0\") " Jul 17 09:28:05.501293 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500905 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "dc51f866-4e1b-4026-976e-bcd8382800a0" (UID: "dc51f866-4e1b-4026-976e-bcd8382800a0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 09:28:05.501293 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.500970 2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-trusted-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:28:05.501293 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.501108 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc51f866-4e1b-4026-976e-bcd8382800a0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "dc51f866-4e1b-4026-976e-bcd8382800a0" (UID: "dc51f866-4e1b-4026-976e-bcd8382800a0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:28:05.501293 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.501199 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "dc51f866-4e1b-4026-976e-bcd8382800a0" (UID: "dc51f866-4e1b-4026-976e-bcd8382800a0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 09:28:05.503248 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.503221 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "dc51f866-4e1b-4026-976e-bcd8382800a0" (UID: "dc51f866-4e1b-4026-976e-bcd8382800a0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:28:05.504007 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.503972 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "dc51f866-4e1b-4026-976e-bcd8382800a0" (UID: "dc51f866-4e1b-4026-976e-bcd8382800a0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:28:05.504007 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.503987 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "dc51f866-4e1b-4026-976e-bcd8382800a0" (UID: "dc51f866-4e1b-4026-976e-bcd8382800a0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 09:28:05.504149 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.504078 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-kube-api-access-zv78f" (OuterVolumeSpecName: "kube-api-access-zv78f") pod "dc51f866-4e1b-4026-976e-bcd8382800a0" (UID: "dc51f866-4e1b-4026-976e-bcd8382800a0"). InnerVolumeSpecName "kube-api-access-zv78f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:28:05.504280 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.504249 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "dc51f866-4e1b-4026-976e-bcd8382800a0" (UID: "dc51f866-4e1b-4026-976e-bcd8382800a0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 09:28:05.602033 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.601997 2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-bound-sa-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:28:05.602033 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.602032 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv78f\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-kube-api-access-zv78f\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:28:05.602289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.602048 2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:28:05.602289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.602063 2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dc51f866-4e1b-4026-976e-bcd8382800a0-registry-certificates\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:28:05.602289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.602077 2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-installation-pull-secrets\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:28:05.602289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.602089 2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dc51f866-4e1b-4026-976e-bcd8382800a0-image-registry-private-configuration\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:28:05.602289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.602103 2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dc51f866-4e1b-4026-976e-bcd8382800a0-ca-trust-extracted\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:28:05.804707 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.804090 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:28:05.816077 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.815811 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26bf1378-169a-44ee-b4a6-3b8116c5cd52-metrics-tls\") pod \"dns-default-6xm7j\" (UID: \"26bf1378-169a-44ee-b4a6-3b8116c5cd52\") " pod="openshift-dns/dns-default-6xm7j" Jul 17 09:28:05.982046 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.981813 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p8vfs\"" Jul 17 09:28:05.990294 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:05.990016 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6xm7j" Jul 17 09:28:06.209472 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.209317 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6xm7j"] Jul 17 09:28:06.216708 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:06.216675 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bf1378_169a_44ee_b4a6_3b8116c5cd52.slice/crio-2955fbdd0f9a535862a17e340ad1537be59816d5987bae94e8e979c69c34674d WatchSource:0}: Error finding container 2955fbdd0f9a535862a17e340ad1537be59816d5987bae94e8e979c69c34674d: Status 404 returned error can't find the container with id 2955fbdd0f9a535862a17e340ad1537be59816d5987bae94e8e979c69c34674d Jul 17 09:28:06.409904 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.409748 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xm7j" event={"ID":"26bf1378-169a-44ee-b4a6-3b8116c5cd52","Type":"ContainerStarted","Data":"2955fbdd0f9a535862a17e340ad1537be59816d5987bae94e8e979c69c34674d"} Jul 17 09:28:06.412298 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.412224 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" event={"ID":"9ce68b72-471e-40be-a4cf-0c493ccfcca7","Type":"ContainerStarted","Data":"6fd736bb27d2f2c66bda179a718d960cab1d6930bb6b66d37faf5ebad208dfea"} Jul 17 09:28:06.412298 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.412278 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" event={"ID":"9ce68b72-471e-40be-a4cf-0c493ccfcca7","Type":"ContainerStarted","Data":"df6c34713fa5543580335300baf854e3042f91a9c8003ea59935382f46775444"} Jul 17 09:28:06.412996 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.412976 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:06.417860 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.417837 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76898c686b-x7dz5" Jul 17 09:28:06.418796 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.418772 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb9ns" event={"ID":"62e2a6ea-beae-40ea-a730-c32f3b172707","Type":"ContainerStarted","Data":"fc2460eac67bc5c3c3357343b591f32117a3600355687e1fa7e9633085563eb3"} Jul 17 09:28:06.418875 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.418821 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vb9ns" event={"ID":"62e2a6ea-beae-40ea-a730-c32f3b172707","Type":"ContainerStarted","Data":"bfde12a7a30ce64a77633e6fc467bf5ddfb9f5e35a601b70bfc6dca995e4bd3c"} Jul 17 09:28:06.432081 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.432042 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" podStartSLOduration=2.43202959 podStartE2EDuration="2.43202959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:06.431527249 +0000 UTC m=+68.970726217" watchObservedRunningTime="2026-07-17 09:28:06.43202959 +0000 UTC m=+68.971228546" Jul 17 09:28:06.463369 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.463334 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76898c686b-x7dz5"] Jul 17 09:28:06.468061 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.468002 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76898c686b-x7dz5"] Jul 17 09:28:06.483415 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.483335 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vb9ns" podStartSLOduration=67.147207978 podStartE2EDuration="1m8.483312676s" totalImagesPullingTime="1.336104698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:06.482378838 +0000 UTC m=+69.021577805" watchObservedRunningTime="2026-07-17 09:28:06.483312676 +0000 UTC m=+69.022511642" Jul 17 09:28:06.816602 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.816493 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:28:06.821255 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:06.821199 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1ed006d9-ebc3-4e86-b54a-4e28aa293ba9-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-svxdt\" (UID: \"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:28:07.004304 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:07.004271 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wwhtg\"" Jul 17 09:28:07.012781 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:07.012751 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" Jul 17 09:28:07.998929 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:07.998895 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc51f866-4e1b-4026-976e-bcd8382800a0" path="/var/lib/kubelet/pods/dc51f866-4e1b-4026-976e-bcd8382800a0/volumes" Jul 17 09:28:12.609298 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:12.609252 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt"] Jul 17 09:28:12.613603 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:12.613387 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed006d9_ebc3_4e86_b54a_4e28aa293ba9.slice/crio-4d21b3021bd58882d15ab38049cffe9eafbedcac27e67c2bdaf6e14a42401ec5 WatchSource:0}: Error finding container 4d21b3021bd58882d15ab38049cffe9eafbedcac27e67c2bdaf6e14a42401ec5: Status 404 returned error can't find the container with id 4d21b3021bd58882d15ab38049cffe9eafbedcac27e67c2bdaf6e14a42401ec5 Jul 17 09:28:12.969929 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:12.969889 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:28:12.972280 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:12.972261 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e3ef95a-3fc4-4565-8c77-df73139c624f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-477zg\" (UID: \"7e3ef95a-3fc4-4565-8c77-df73139c624f\") " pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:28:13.243698 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.243622 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gmxxn\"" Jul 17 09:28:13.251915 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.251886 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-477zg" Jul 17 09:28:13.387046 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.387009 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-477zg"] Jul 17 09:28:13.390105 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:13.390070 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3ef95a_3fc4_4565_8c77_df73139c624f.slice/crio-b9753e93799d2defa6ae988d6a01f81effa56e58578dd9951c55eefd8f0a3c2c WatchSource:0}: Error finding container b9753e93799d2defa6ae988d6a01f81effa56e58578dd9951c55eefd8f0a3c2c: Status 404 returned error can't find the container with id b9753e93799d2defa6ae988d6a01f81effa56e58578dd9951c55eefd8f0a3c2c Jul 17 09:28:13.445698 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.445661 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-477zg" event={"ID":"7e3ef95a-3fc4-4565-8c77-df73139c624f","Type":"ContainerStarted","Data":"b9753e93799d2defa6ae988d6a01f81effa56e58578dd9951c55eefd8f0a3c2c"} Jul 17 09:28:13.447245 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.447178 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" event={"ID":"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9","Type":"ContainerStarted","Data":"4d21b3021bd58882d15ab38049cffe9eafbedcac27e67c2bdaf6e14a42401ec5"} Jul 17 09:28:13.449724 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.449691 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xm7j" event={"ID":"26bf1378-169a-44ee-b4a6-3b8116c5cd52","Type":"ContainerStarted","Data":"2dedd1df7009fb7e45208e759b0ed0d49f8bb298f70ea9d609c0b55ea26937d8"} Jul 17 09:28:13.449849 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.449740 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6xm7j" event={"ID":"26bf1378-169a-44ee-b4a6-3b8116c5cd52","Type":"ContainerStarted","Data":"b5b08b93d4a55938bffb559667f441fbfecf0e486ab88ee440f6f27d75d40386"} Jul 17 09:28:13.449849 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.449794 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6xm7j" Jul 17 09:28:13.451612 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.451580 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s6khr" event={"ID":"1d34fb73-3a61-49ce-be61-32b998fa34f1","Type":"ContainerStarted","Data":"0ff205dfebc049c17023d257500a66b22da667d6102352607ffbcdc6f2aedb1b"} Jul 17 09:28:13.451735 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.451635 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:28:13.453089 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.453066 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" event={"ID":"7c46b1c8-d9ec-475a-9ce7-37264a5b467c","Type":"ContainerStarted","Data":"bed178e8babd8b03dd245af8b9d3600eb2fd8ccb880115f2517322effeba3bd5"} Jul 17 09:28:13.453236 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.453220 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:13.454787 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.454766 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pkvmq" event={"ID":"ff2f29e6-ee5e-433c-a625-30090f8f5d9c","Type":"ContainerStarted","Data":"77ba4282ba5e698739c31b5e37431ff860bc788dd23623383bc71bb1edd62fd5"} Jul 17 09:28:13.455264 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.455231 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" Jul 17 09:28:13.468979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.468934 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6xm7j" podStartSLOduration=34.248993995 podStartE2EDuration="40.468919758s" totalImagesPullingTime="6.219925763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:13.468618814 +0000 UTC m=+76.007817817" watchObservedRunningTime="2026-07-17 09:28:13.468919758 +0000 UTC m=+76.008118727" Jul 17 09:28:13.484356 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.484297 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s6khr" podStartSLOduration=67.311434738 podStartE2EDuration="1m15.48427521s" totalImagesPullingTime="8.172840472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:13.483998124 +0000 UTC m=+76.023197091" watchObservedRunningTime="2026-07-17 09:28:13.48427521 +0000 UTC m=+76.023474177" Jul 17 09:28:13.501821 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:13.501727 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pkvmq" podStartSLOduration=67.032618856 podStartE2EDuration="1m15.501710234s" totalImagesPullingTime="8.469091378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:13.501298903 +0000 UTC m=+76.040497870" watchObservedRunningTime="2026-07-17 09:28:13.501710234 +0000 UTC m=+76.040909200" Jul 17 09:28:14.463825 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:14.463756 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" event={"ID":"1ed006d9-ebc3-4e86-b54a-4e28aa293ba9","Type":"ContainerStarted","Data":"b60f254b80672d1dc3e1460969120e4ee9bcc5d1c151dcd14fb816e63394b4cf"} Jul 17 09:28:14.481184 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:14.481129 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-658858dd96-7z45j" podStartSLOduration=3.274114654 podStartE2EDuration="10.481110659s" totalImagesPullingTime="7.206996005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:13.515593885 +0000 UTC m=+76.054792855" watchObservedRunningTime="2026-07-17 09:28:14.481110659 +0000 UTC m=+77.020309626" Jul 17 09:28:14.482289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:14.482243 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-svxdt" podStartSLOduration=38.830900684 podStartE2EDuration="40.482226764s" totalImagesPullingTime="1.65132608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:14.480671023 +0000 UTC m=+77.019869987" watchObservedRunningTime="2026-07-17 09:28:14.482226764 +0000 UTC m=+77.021425734" Jul 17 09:28:19.483936 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:19.483796 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-477zg" event={"ID":"7e3ef95a-3fc4-4565-8c77-df73139c624f","Type":"ContainerStarted","Data":"956bdc63fc384d55b76eeb837260360fb253b972a0a11b5b7a1df4e81bf12a74"} Jul 17 09:28:22.497126 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:22.497038 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-477zg" event={"ID":"7e3ef95a-3fc4-4565-8c77-df73139c624f","Type":"ContainerStarted","Data":"4f7cf5ecae85a9364bc43f23ce6e33fc995a1d6b831d3fc86f5bf749e6865d4a"} Jul 17 09:28:23.466558 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:23.466526 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6xm7j" Jul 17 09:28:25.149116 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:25.149068 2584 patch_prober.go:28] interesting pod/image-registry-7675b5767f-wp4r5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 17 09:28:25.149561 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:25.149144 2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" podUID="9ce68b72-471e-40be-a4cf-0c493ccfcca7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 09:28:26.514884 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:26.514849 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-477zg" event={"ID":"7e3ef95a-3fc4-4565-8c77-df73139c624f","Type":"ContainerStarted","Data":"c0a4d3989f5a3423e9a2c307e45b80fba43a9e59182ecc3b3053ab1cc8a5255d"} Jul 17 09:28:28.430262 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:28.430236 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7675b5767f-wp4r5" Jul 17 09:28:28.451952 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:28.451901 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-477zg" podStartSLOduration=35.179592287 podStartE2EDuration="47.451886532s" totalImagesPullingTime="12.272294245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 09:27:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:26.532321071 +0000 UTC m=+89.071520038" watchObservedRunningTime="2026-07-17 09:28:28.451886532 +0000 UTC m=+90.991085497" Jul 17 09:28:40.197694 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.197661 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vsccb"] Jul 17 09:28:40.203468 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.203442 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.206029 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.206007 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 09:28:40.206240 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.206227 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ddkt4\"" Jul 17 09:28:40.206499 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.206481 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 09:28:40.206600 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.206511 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 09:28:40.207492 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.207476 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 09:28:40.212184 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.212165 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j2jz4"] Jul 17 09:28:40.215329 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.215312 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vsccb"] Jul 17 09:28:40.215433 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.215421 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.217931 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.217825 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 09:28:40.217931 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.217867 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 09:28:40.217931 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.217900 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pf7jj\"" Jul 17 09:28:40.218151 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.218135 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 09:28:40.272792 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272762 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-metrics-client-ca\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.272943 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272811 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbmh\" (UniqueName: \"kubernetes.io/projected/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-kube-api-access-2pbmh\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.272943 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272831 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.272943 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272853 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.272943 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272891 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kl2d\" (UniqueName: \"kubernetes.io/projected/251db854-1393-40ee-8cd5-0469a9ea752e-kube-api-access-4kl2d\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.272943 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272911 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-textfile\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.272943 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272929 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/251db854-1393-40ee-8cd5-0469a9ea752e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.273133 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272954 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-root\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.273133 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272975 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.273133 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.272991 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-sys\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.273133 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.273007 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.273133 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.273030 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.273133 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.273086 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251db854-1393-40ee-8cd5-0469a9ea752e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.273133 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.273111 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-wtmp\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.273343 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.273137 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-tls\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374358 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374320 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251db854-1393-40ee-8cd5-0469a9ea752e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.374358 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374357 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-wtmp\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374548 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374376 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-tls\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374548 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374417 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-metrics-client-ca\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374548 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374457 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbmh\" (UniqueName: \"kubernetes.io/projected/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-kube-api-access-2pbmh\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374548 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374482 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374548 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374500 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374548 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374524 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kl2d\" (UniqueName: \"kubernetes.io/projected/251db854-1393-40ee-8cd5-0469a9ea752e-kube-api-access-4kl2d\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.374548 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:28:40.374527 2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374557 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-textfile\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374569 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-wtmp\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374582 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/251db854-1393-40ee-8cd5-0469a9ea752e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374622 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-root\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374648 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374675 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-sys\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374697 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374722 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:28:40.374801 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-tls podName:9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7 nodeName:}" failed. No retries permitted until 2026-07-17 09:28:40.874777164 +0000 UTC m=+103.413976110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-tls") pod "node-exporter-j2jz4" (UID: "9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7") : secret "node-exporter-tls" not found Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374847 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-sys\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:28:40.374873 2584 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:28:40.374950 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-tls podName:251db854-1393-40ee-8cd5-0469a9ea752e nodeName:}" failed. No retries permitted until 2026-07-17 09:28:40.874933095 +0000 UTC m=+103.414132051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-vsccb" (UID: "251db854-1393-40ee-8cd5-0469a9ea752e") : secret "kube-state-metrics-tls" not found Jul 17 09:28:40.374962 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.374954 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-root\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.375629 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.375097 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-metrics-client-ca\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.375629 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.375229 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-textfile\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.375629 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.375311 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/251db854-1393-40ee-8cd5-0469a9ea752e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.375629 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.375440 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.375629 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.375591 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/251db854-1393-40ee-8cd5-0469a9ea752e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.375855 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.375761 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.377329 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.377312 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.377547 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.377527 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.390332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.390309 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbmh\" (UniqueName: \"kubernetes.io/projected/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-kube-api-access-2pbmh\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.390772 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.390754 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kl2d\" (UniqueName: \"kubernetes.io/projected/251db854-1393-40ee-8cd5-0469a9ea752e-kube-api-access-4kl2d\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.879232 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.879194 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-tls\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.879470 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.879306 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:40.881753 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.881719 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7-node-exporter-tls\") pod \"node-exporter-j2jz4\" (UID: \"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7\") " pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:40.881849 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:40.881758 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/251db854-1393-40ee-8cd5-0469a9ea752e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vsccb\" (UID: \"251db854-1393-40ee-8cd5-0469a9ea752e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:41.112950 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.112915 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" Jul 17 09:28:41.124151 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.124115 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2jz4" Jul 17 09:28:41.137245 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:41.137214 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef4032e_c364_4ac6_b6a0_fbf1e499d2d7.slice/crio-44bc8b161ee5d9317646fcfd950ea56708780ec3066298137e878eb28ed819da WatchSource:0}: Error finding container 44bc8b161ee5d9317646fcfd950ea56708780ec3066298137e878eb28ed819da: Status 404 returned error can't find the container with id 44bc8b161ee5d9317646fcfd950ea56708780ec3066298137e878eb28ed819da Jul 17 09:28:41.247478 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.247446 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 09:28:41.252946 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.252920 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.254209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.254169 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vsccb"] Jul 17 09:28:41.255424 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.255310 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 09:28:41.256288 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.256267 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 09:28:41.256379 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.256339 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 09:28:41.256622 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.256588 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 09:28:41.256726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.256634 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rlmgt\"" Jul 17 09:28:41.256787 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.256729 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 09:28:41.256903 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.256886 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 09:28:41.257904 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.257709 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 09:28:41.257904 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.257819 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 09:28:41.257904 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.257837 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 09:28:41.258368 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:41.258345 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251db854_1393_40ee_8cd5_0469a9ea752e.slice/crio-6a9770b70431dd397c2ecf9b756670aa0566dc651acd2bfc57bbb62a5c0a7eca WatchSource:0}: Error finding container 6a9770b70431dd397c2ecf9b756670aa0566dc651acd2bfc57bbb62a5c0a7eca: Status 404 returned error can't find the container with id 6a9770b70431dd397c2ecf9b756670aa0566dc651acd2bfc57bbb62a5c0a7eca Jul 17 09:28:41.265364 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.265346 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 09:28:41.282707 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282678 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.282852 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282715 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.282852 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282733 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.282852 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282750 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6fe5847-5268-43e8-9d78-1f59b27f57d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.282852 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282832 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.283052 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282868 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.283052 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282886 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6fe5847-5268-43e8-9d78-1f59b27f57d8-config-out\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.283052 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282943 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.283052 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.282979 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.283052 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.283001 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.283052 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.283033 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.283335 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.283055 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-web-config\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.283335 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.283079 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgt9x\" (UniqueName: \"kubernetes.io/projected/a6fe5847-5268-43e8-9d78-1f59b27f57d8-kube-api-access-lgt9x\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.384465 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384357 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.384465 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384414 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6fe5847-5268-43e8-9d78-1f59b27f57d8-config-out\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.384465 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384441 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384468 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384491 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384537 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384563 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-web-config\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:28:41.384603 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-trusted-ca-bundle podName:a6fe5847-5268-43e8-9d78-1f59b27f57d8 nodeName:}" failed. No retries permitted until 2026-07-17 09:28:41.884577657 +0000 UTC m=+104.423776615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "a6fe5847-5268-43e8-9d78-1f59b27f57d8") : configmap references non-existent config key: ca-bundle.crt Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384632 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgt9x\" (UniqueName: \"kubernetes.io/projected/a6fe5847-5268-43e8-9d78-1f59b27f57d8-kube-api-access-lgt9x\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384688 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384725 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384757 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384784 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6fe5847-5268-43e8-9d78-1f59b27f57d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.384838 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.385635 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.385822 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.387451 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6fe5847-5268-43e8-9d78-1f59b27f57d8-config-out\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.388215 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.388615 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-web-config\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389920 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.389423 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.389920 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.389490 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.390014 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.389934 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.390537 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.390514 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.390925 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.390907 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6fe5847-5268-43e8-9d78-1f59b27f57d8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.391221 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.391203 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6fe5847-5268-43e8-9d78-1f59b27f57d8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.394921 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.394902 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgt9x\" (UniqueName: \"kubernetes.io/projected/a6fe5847-5268-43e8-9d78-1f59b27f57d8-kube-api-access-lgt9x\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.570890 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.570759 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2jz4" event={"ID":"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7","Type":"ContainerStarted","Data":"44bc8b161ee5d9317646fcfd950ea56708780ec3066298137e878eb28ed819da"} Jul 17 09:28:41.572196 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.572164 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" event={"ID":"251db854-1393-40ee-8cd5-0469a9ea752e","Type":"ContainerStarted","Data":"6a9770b70431dd397c2ecf9b756670aa0566dc651acd2bfc57bbb62a5c0a7eca"} Jul 17 09:28:41.890252 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.890218 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:41.891149 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:41.891127 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6fe5847-5268-43e8-9d78-1f59b27f57d8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6fe5847-5268-43e8-9d78-1f59b27f57d8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:42.166552 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:42.166519 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 09:28:42.578800 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:42.578741 2584 generic.go:356] "Generic (PLEG): container finished" podID="9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7" containerID="8a2c866e50391614a15b8f1e3632b3414f2957b8303125360d7ecfe437480168" exitCode=0 Jul 17 09:28:42.578800 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:42.578780 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2jz4" event={"ID":"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7","Type":"ContainerDied","Data":"8a2c866e50391614a15b8f1e3632b3414f2957b8303125360d7ecfe437480168"} Jul 17 09:28:42.633601 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:42.633573 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 09:28:42.636360 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:42.636334 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fe5847_5268_43e8_9d78_1f59b27f57d8.slice/crio-d41fe6b0ec2bff2f7128e9c63eb7bf204d48ccd492356d897b7c8a1ab2f25bd4 WatchSource:0}: Error finding container d41fe6b0ec2bff2f7128e9c63eb7bf204d48ccd492356d897b7c8a1ab2f25bd4: Status 404 returned error can't find the container with id d41fe6b0ec2bff2f7128e9c63eb7bf204d48ccd492356d897b7c8a1ab2f25bd4 Jul 17 09:28:43.172545 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.172508 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b5884db65-tjrzh"] Jul 17 09:28:43.189760 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.189729 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.190474 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.190451 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b5884db65-tjrzh"] Jul 17 09:28:43.192282 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.192261 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 09:28:43.192411 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.192371 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 09:28:43.192535 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.192495 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 09:28:43.193714 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.193068 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3kqeo6lmhna2s\"" Jul 17 09:28:43.193714 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.193237 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 09:28:43.193714 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.193295 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pd7qf\"" Jul 17 09:28:43.197776 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.194356 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 09:28:43.203646 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.203622 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wbcc\" (UniqueName: \"kubernetes.io/projected/e1c089c0-767a-407d-b788-7fde202cba7d-kube-api-access-5wbcc\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.203734 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.203658 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c089c0-767a-407d-b788-7fde202cba7d-metrics-client-ca\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.203734 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.203694 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.203734 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.203720 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.203848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.203751 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.203848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.203769 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.203848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.203790 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-grpc-tls\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.203848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.203819 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-tls\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.304492 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.304456 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.304492 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.304496 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.304728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.304523 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-grpc-tls\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.304728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.304580 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-tls\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.304728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.304614 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wbcc\" (UniqueName: \"kubernetes.io/projected/e1c089c0-767a-407d-b788-7fde202cba7d-kube-api-access-5wbcc\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.304728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.304645 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c089c0-767a-407d-b788-7fde202cba7d-metrics-client-ca\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.304728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.304693 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.304728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.304719 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.305663 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.305633 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1c089c0-767a-407d-b788-7fde202cba7d-metrics-client-ca\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.307586 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.307552 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.307728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.307705 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.307938 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.307919 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-tls\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.308048 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.308026 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.308146 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.308128 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.308242 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.308225 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1c089c0-767a-407d-b788-7fde202cba7d-secret-grpc-tls\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.312811 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.312792 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wbcc\" (UniqueName: \"kubernetes.io/projected/e1c089c0-767a-407d-b788-7fde202cba7d-kube-api-access-5wbcc\") pod \"thanos-querier-b5884db65-tjrzh\" (UID: \"e1c089c0-767a-407d-b788-7fde202cba7d\") " pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.502843 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.502759 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:43.585444 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.585381 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2jz4" event={"ID":"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7","Type":"ContainerStarted","Data":"a67b45f1f23fdf30f547c4b9dc8005088934cbccb053c08c87734cb12469e8f9"} Jul 17 09:28:43.585904 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.585507 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2jz4" event={"ID":"9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7","Type":"ContainerStarted","Data":"3048216e5cd692c6480e7ebf37e74ab0a9255ba07ce1a03a22529b450a6d1cbc"} Jul 17 09:28:43.587874 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.587846 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" event={"ID":"251db854-1393-40ee-8cd5-0469a9ea752e","Type":"ContainerStarted","Data":"b7c183f5ffc0500a39bea5790ff66f428b9d3ccfc4c2bf279033cf04172ddc43"} Jul 17 09:28:43.588018 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.587910 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" event={"ID":"251db854-1393-40ee-8cd5-0469a9ea752e","Type":"ContainerStarted","Data":"678503d07f6f904484eaf429a16afcd309dea93b20e7ceed7fab84eaddc9a8a1"} Jul 17 09:28:43.588018 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.587941 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" event={"ID":"251db854-1393-40ee-8cd5-0469a9ea752e","Type":"ContainerStarted","Data":"9a1ef498781fc3443fa538185437c59a7b25490a18b1ce7eed70a70df345b576"} Jul 17 09:28:43.589285 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.589253 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6fe5847-5268-43e8-9d78-1f59b27f57d8","Type":"ContainerStarted","Data":"d41fe6b0ec2bff2f7128e9c63eb7bf204d48ccd492356d897b7c8a1ab2f25bd4"} Jul 17 09:28:43.610230 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.610176 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j2jz4" podStartSLOduration=2.545362062 podStartE2EDuration="3.610159348s" totalImagesPullingTime="988.789874ms" totalInitContainerRuntime="76.007412ms" isStatefulPod=true podCreationTimestamp="2026-07-17 09:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:43.607756328 +0000 UTC m=+106.146955297" watchObservedRunningTime="2026-07-17 09:28:43.610159348 +0000 UTC m=+106.149358315" Jul 17 09:28:43.630170 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.630121 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-vsccb" podStartSLOduration=2.350649388 podStartE2EDuration="3.630103865s" totalImagesPullingTime="1.279454477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:43.628634848 +0000 UTC m=+106.167833815" watchObservedRunningTime="2026-07-17 09:28:43.630103865 +0000 UTC m=+106.169302833" Jul 17 09:28:43.873377 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:43.873343 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b5884db65-tjrzh"] Jul 17 09:28:43.980226 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:28:43.980192 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c089c0_767a_407d_b788_7fde202cba7d.slice/crio-f43612bbcd564866d1420f43c46405e8178928a15474f76ed9ef68a1de78b005 WatchSource:0}: Error finding container f43612bbcd564866d1420f43c46405e8178928a15474f76ed9ef68a1de78b005: Status 404 returned error can't find the container with id f43612bbcd564866d1420f43c46405e8178928a15474f76ed9ef68a1de78b005 Jul 17 09:28:44.467250 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:44.467218 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s6khr" Jul 17 09:28:44.594638 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:44.594604 2584 generic.go:356] "Generic (PLEG): container finished" podID="a6fe5847-5268-43e8-9d78-1f59b27f57d8" containerID="d80bcc45b480b32b0016f1cb4fd0c743bf56a66924ba821002a018252f3be0c1" exitCode=0 Jul 17 09:28:44.595059 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:44.594680 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6fe5847-5268-43e8-9d78-1f59b27f57d8","Type":"ContainerDied","Data":"d80bcc45b480b32b0016f1cb4fd0c743bf56a66924ba821002a018252f3be0c1"} Jul 17 09:28:44.595987 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:44.595960 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" event={"ID":"e1c089c0-767a-407d-b788-7fde202cba7d","Type":"ContainerStarted","Data":"f43612bbcd564866d1420f43c46405e8178928a15474f76ed9ef68a1de78b005"} Jul 17 09:28:47.611349 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:47.611320 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6fe5847-5268-43e8-9d78-1f59b27f57d8","Type":"ContainerStarted","Data":"ca90007e1124d17f8c5aa945a50a73389871c24cff3813576fe9f62eba94f10a"} Jul 17 09:28:47.611676 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:47.611392 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6fe5847-5268-43e8-9d78-1f59b27f57d8","Type":"ContainerStarted","Data":"396cd7be498c70bea9ddfbb5a10d3e52397483ebac716c8c00db8e01832de944"} Jul 17 09:28:47.611676 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:47.611438 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6fe5847-5268-43e8-9d78-1f59b27f57d8","Type":"ContainerStarted","Data":"250ea2f635cfcf48e0a2fb714174622fcfa513c37b0d0d961ff3f3901ba615ab"} Jul 17 09:28:47.611676 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:47.611472 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6fe5847-5268-43e8-9d78-1f59b27f57d8","Type":"ContainerStarted","Data":"c08a989768c6317368932419a7e5778a6f85b10a1e00b5333158dceb293087d5"} Jul 17 09:28:47.611676 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:47.611501 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6fe5847-5268-43e8-9d78-1f59b27f57d8","Type":"ContainerStarted","Data":"70ed06c73beb47b87128644b0abf08100e1bedc206db443b9087ea3b7ee158c5"} Jul 17 09:28:47.613791 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:47.613769 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" event={"ID":"e1c089c0-767a-407d-b788-7fde202cba7d","Type":"ContainerStarted","Data":"6e170d6cc530ecf29b577d12ababe321a1976778feebf39d251c8513fdfbef56"} Jul 17 09:28:47.613887 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:47.613827 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" event={"ID":"e1c089c0-767a-407d-b788-7fde202cba7d","Type":"ContainerStarted","Data":"162a0d2c4f217d69404c73fc392ce118dce75a60aca3f915977bf5be2aee76e4"} Jul 17 09:28:47.613887 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:47.613853 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" event={"ID":"e1c089c0-767a-407d-b788-7fde202cba7d","Type":"ContainerStarted","Data":"09d02a3cbd1a5000a142491dd0010a720db874386e5ab6ec8dbc600d4c666867"} Jul 17 09:28:49.625153 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:49.625115 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6fe5847-5268-43e8-9d78-1f59b27f57d8","Type":"ContainerStarted","Data":"d951c382356b742eeeafecf9b303ebafd2c298e02638814a80cc2182ab05f25f"} Jul 17 09:28:49.627829 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:49.627800 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" event={"ID":"e1c089c0-767a-407d-b788-7fde202cba7d","Type":"ContainerStarted","Data":"fe69ef0f3ee1689e520349d7f64266d032f3ca05299828f198d50e498bb61d55"} Jul 17 09:28:49.627947 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:49.627856 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" event={"ID":"e1c089c0-767a-407d-b788-7fde202cba7d","Type":"ContainerStarted","Data":"3305aba6af39d891a4e29c5c2146b9886c6860ed2d54271852d3b362ae811854"} Jul 17 09:28:49.627947 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:49.627882 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" event={"ID":"e1c089c0-767a-407d-b788-7fde202cba7d","Type":"ContainerStarted","Data":"a072f88b2565e36d4a1f31f8c07547a1d7496ca48fe5502267ff7449c9ffd88a"} Jul 17 09:28:49.627947 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:49.627914 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:49.652084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:49.652024 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.828054105 podStartE2EDuration="8.651992342s" totalImagesPullingTime="4.799069965s" totalInitContainerRuntime="24.868272ms" isStatefulPod=false podCreationTimestamp="2026-07-17 09:28:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:49.650581351 +0000 UTC m=+112.189780327" watchObservedRunningTime="2026-07-17 09:28:49.651992342 +0000 UTC m=+112.191191304" Jul 17 09:28:49.672426 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:49.672362 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" podStartSLOduration=2.408540654 podStartE2EDuration="6.672345935s" totalImagesPullingTime="4.263805281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:28:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:28:49.671234905 +0000 UTC m=+112.210433871" watchObservedRunningTime="2026-07-17 09:28:49.672345935 +0000 UTC m=+112.211544923" Jul 17 09:28:55.639143 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:55.639121 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b5884db65-tjrzh" Jul 17 09:28:55.653448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:55.653417 2584 generic.go:356] "Generic (PLEG): container finished" podID="b8c24f15-a691-4a51-a970-a00697042f6e" containerID="8d6fc0ce569824578856be053337bf600f3a0c684d62fc782778679a3c965e6d" exitCode=0 Jul 17 09:28:55.653562 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:55.653489 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kb2fs" event={"ID":"b8c24f15-a691-4a51-a970-a00697042f6e","Type":"ContainerDied","Data":"8d6fc0ce569824578856be053337bf600f3a0c684d62fc782778679a3c965e6d"} Jul 17 09:28:55.653815 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:55.653800 2584 scope.go:117] "RemoveContainer" containerID="8d6fc0ce569824578856be053337bf600f3a0c684d62fc782778679a3c965e6d" Jul 17 09:28:56.659009 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:56.658975 2584 generic.go:356] "Generic (PLEG): container finished" podID="f0903913-3b04-448f-b22d-c24a066ad245" containerID="f8948fa708d1dd996ade6043fd7f03a88b9c8fe35c38b0d131fc717cc2019b86" exitCode=0 Jul 17 09:28:56.659497 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:56.659049 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" event={"ID":"f0903913-3b04-448f-b22d-c24a066ad245","Type":"ContainerDied","Data":"f8948fa708d1dd996ade6043fd7f03a88b9c8fe35c38b0d131fc717cc2019b86"} Jul 17 09:28:56.659497 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:56.659448 2584 scope.go:117] "RemoveContainer" containerID="f8948fa708d1dd996ade6043fd7f03a88b9c8fe35c38b0d131fc717cc2019b86" Jul 17 09:28:56.660992 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:56.660903 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-kb2fs" event={"ID":"b8c24f15-a691-4a51-a970-a00697042f6e","Type":"ContainerStarted","Data":"3414710f1cddd547ade1c5cb6b244b1c42d8f0100967bd440e3e88924eb91d3f"} Jul 17 09:28:57.667034 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:28:57.666997 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-frxhg" event={"ID":"f0903913-3b04-448f-b22d-c24a066ad245","Type":"ContainerStarted","Data":"d8e05105795c832c706fb57b013fcf78f259ef89646eed7c2227abb519a1cc66"} Jul 17 09:30:50.385467 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.385427 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8"] Jul 17 09:30:50.388610 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.388593 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.391391 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.391362 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 09:30:50.391624 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.391550 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-79wp7\"" Jul 17 09:30:50.392586 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.392560 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 09:30:50.398086 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.398057 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8"] Jul 17 09:30:50.489565 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.489530 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.489729 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.489582 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62zx\" (UniqueName: \"kubernetes.io/projected/74dfe29e-8c92-4781-a869-e7f9091203e7-kube-api-access-f62zx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.489729 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.489660 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.590211 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.590179 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.590354 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.590242 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.590354 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.590289 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62zx\" (UniqueName: \"kubernetes.io/projected/74dfe29e-8c92-4781-a869-e7f9091203e7-kube-api-access-f62zx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.590675 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.590654 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.590711 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.590662 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.598581 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.598550 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62zx\" (UniqueName: \"kubernetes.io/projected/74dfe29e-8c92-4781-a869-e7f9091203e7-kube-api-access-f62zx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.698862 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.698775 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:30:50.823348 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:50.823282 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8"] Jul 17 09:30:50.826107 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:30:50.826081 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74dfe29e_8c92_4781_a869_e7f9091203e7.slice/crio-69d4a9d3720dc972f4e500f75529a5095b0c057b89f229f9c3d8106bd84398a3 WatchSource:0}: Error finding container 69d4a9d3720dc972f4e500f75529a5095b0c057b89f229f9c3d8106bd84398a3: Status 404 returned error can't find the container with id 69d4a9d3720dc972f4e500f75529a5095b0c057b89f229f9c3d8106bd84398a3 Jul 17 09:30:51.094872 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:51.094839 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" event={"ID":"74dfe29e-8c92-4781-a869-e7f9091203e7","Type":"ContainerStarted","Data":"69d4a9d3720dc972f4e500f75529a5095b0c057b89f229f9c3d8106bd84398a3"} Jul 17 09:30:58.123710 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:58.123678 2584 generic.go:356] "Generic (PLEG): container finished" podID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerID="c78c1fd1b57f5438c1c2ed7e513d1a4d7f72e4d0f49fa24a0983f2e1664f8f44" exitCode=0 Jul 17 09:30:58.124124 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:30:58.123768 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" event={"ID":"74dfe29e-8c92-4781-a869-e7f9091203e7","Type":"ContainerDied","Data":"c78c1fd1b57f5438c1c2ed7e513d1a4d7f72e4d0f49fa24a0983f2e1664f8f44"} Jul 17 09:31:00.133228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:00.133200 2584 generic.go:356] "Generic (PLEG): container finished" podID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerID="db8bc0635ca43d94eae173763a221227c561b6c43a6caa5e9d7786d428df1dd2" exitCode=0 Jul 17 09:31:00.133581 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:00.133273 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" event={"ID":"74dfe29e-8c92-4781-a869-e7f9091203e7","Type":"ContainerDied","Data":"db8bc0635ca43d94eae173763a221227c561b6c43a6caa5e9d7786d428df1dd2"} Jul 17 09:31:07.167184 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:07.167104 2584 generic.go:356] "Generic (PLEG): container finished" podID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerID="f873ee4af5b99bc7aefdaf6824c89f61f5384e85dff1d9c4ee98c423acd1ba6d" exitCode=0 Jul 17 09:31:07.167548 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:07.167191 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" event={"ID":"74dfe29e-8c92-4781-a869-e7f9091203e7","Type":"ContainerDied","Data":"f873ee4af5b99bc7aefdaf6824c89f61f5384e85dff1d9c4ee98c423acd1ba6d"} Jul 17 09:31:08.293802 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.293776 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:31:08.326543 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.326516 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f62zx\" (UniqueName: \"kubernetes.io/projected/74dfe29e-8c92-4781-a869-e7f9091203e7-kube-api-access-f62zx\") pod \"74dfe29e-8c92-4781-a869-e7f9091203e7\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " Jul 17 09:31:08.326684 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.326554 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-bundle\") pod \"74dfe29e-8c92-4781-a869-e7f9091203e7\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " Jul 17 09:31:08.326747 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.326689 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-util\") pod \"74dfe29e-8c92-4781-a869-e7f9091203e7\" (UID: \"74dfe29e-8c92-4781-a869-e7f9091203e7\") " Jul 17 09:31:08.327223 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.327202 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-bundle" (OuterVolumeSpecName: "bundle") pod "74dfe29e-8c92-4781-a869-e7f9091203e7" (UID: "74dfe29e-8c92-4781-a869-e7f9091203e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:31:08.328761 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.328735 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dfe29e-8c92-4781-a869-e7f9091203e7-kube-api-access-f62zx" (OuterVolumeSpecName: "kube-api-access-f62zx") pod "74dfe29e-8c92-4781-a869-e7f9091203e7" (UID: "74dfe29e-8c92-4781-a869-e7f9091203e7"). InnerVolumeSpecName "kube-api-access-f62zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:31:08.330953 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.330933 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-util" (OuterVolumeSpecName: "util") pod "74dfe29e-8c92-4781-a869-e7f9091203e7" (UID: "74dfe29e-8c92-4781-a869-e7f9091203e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:31:08.427378 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.427299 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f62zx\" (UniqueName: \"kubernetes.io/projected/74dfe29e-8c92-4781-a869-e7f9091203e7-kube-api-access-f62zx\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:08.427378 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.427324 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:08.427378 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:08.427333 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74dfe29e-8c92-4781-a869-e7f9091203e7-util\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:09.176530 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:09.176499 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" Jul 17 09:31:09.176695 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:09.176494 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540lzwn8" event={"ID":"74dfe29e-8c92-4781-a869-e7f9091203e7","Type":"ContainerDied","Data":"69d4a9d3720dc972f4e500f75529a5095b0c057b89f229f9c3d8106bd84398a3"} Jul 17 09:31:09.176695 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:09.176620 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69d4a9d3720dc972f4e500f75529a5095b0c057b89f229f9c3d8106bd84398a3" Jul 17 09:31:17.330498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.330460 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c"] Jul 17 09:31:17.330967 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.330911 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerName="util" Jul 17 09:31:17.330967 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.330930 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerName="util" Jul 17 09:31:17.330967 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.330942 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerName="pull" Jul 17 09:31:17.330967 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.330950 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerName="pull" Jul 17 09:31:17.330967 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.330966 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerName="extract" Jul 17 09:31:17.331215 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.330974 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerName="extract" Jul 17 09:31:17.331215 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.331068 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="74dfe29e-8c92-4781-a869-e7f9091203e7" containerName="extract" Jul 17 09:31:17.371649 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.371609 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c"] Jul 17 09:31:17.371809 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.371719 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.375899 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.375853 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 09:31:17.376081 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.375993 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 09:31:17.377080 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.377062 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-79wp7\"" Jul 17 09:31:17.508985 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.508942 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfb5g\" (UniqueName: \"kubernetes.io/projected/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-kube-api-access-wfb5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.509156 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.508999 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.509156 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.509077 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.610289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.610256 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfb5g\" (UniqueName: \"kubernetes.io/projected/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-kube-api-access-wfb5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.610482 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.610305 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.610482 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.610349 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.610796 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.610775 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.610832 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.610790 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.622525 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.622498 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfb5g\" (UniqueName: \"kubernetes.io/projected/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-kube-api-access-wfb5g\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.681282 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.681244 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:17.821149 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:17.821108 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c"] Jul 17 09:31:17.824268 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:31:17.824240 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27dd28f_4d0d_46d1_8e0c_d698663db9c3.slice/crio-60fcdfc0a47e68146fa70225463fe1b64d4a4824e09c3c1e12f6d488f4ad6f86 WatchSource:0}: Error finding container 60fcdfc0a47e68146fa70225463fe1b64d4a4824e09c3c1e12f6d488f4ad6f86: Status 404 returned error can't find the container with id 60fcdfc0a47e68146fa70225463fe1b64d4a4824e09c3c1e12f6d488f4ad6f86 Jul 17 09:31:18.211328 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:18.211246 2584 generic.go:356] "Generic (PLEG): container finished" podID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerID="e35820a86fc898a17d650ecbf7f429297f258adb6cad393f7a5bbe1cb62e0eb8" exitCode=0 Jul 17 09:31:18.211533 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:18.211334 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" event={"ID":"c27dd28f-4d0d-46d1-8e0c-d698663db9c3","Type":"ContainerDied","Data":"e35820a86fc898a17d650ecbf7f429297f258adb6cad393f7a5bbe1cb62e0eb8"} Jul 17 09:31:18.211533 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:18.211391 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" event={"ID":"c27dd28f-4d0d-46d1-8e0c-d698663db9c3","Type":"ContainerStarted","Data":"60fcdfc0a47e68146fa70225463fe1b64d4a4824e09c3c1e12f6d488f4ad6f86"} Jul 17 09:31:20.084304 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.084226 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc"] Jul 17 09:31:20.090081 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.090059 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" Jul 17 09:31:20.094913 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.094888 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-p5xw6\"" Jul 17 09:31:20.096319 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.096295 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 09:31:20.096722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.096682 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 09:31:20.101424 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.101378 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc"] Jul 17 09:31:20.232687 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.232649 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4v9q\" (UniqueName: \"kubernetes.io/projected/c135edcd-9eb2-4aaf-b63d-dd5505ce23f4-kube-api-access-r4v9q\") pod \"cert-manager-cainjector-559cdd8c87-bfxhc\" (UID: \"c135edcd-9eb2-4aaf-b63d-dd5505ce23f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" Jul 17 09:31:20.232838 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.232716 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c135edcd-9eb2-4aaf-b63d-dd5505ce23f4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bfxhc\" (UID: \"c135edcd-9eb2-4aaf-b63d-dd5505ce23f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" Jul 17 09:31:20.333955 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.333905 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4v9q\" (UniqueName: \"kubernetes.io/projected/c135edcd-9eb2-4aaf-b63d-dd5505ce23f4-kube-api-access-r4v9q\") pod \"cert-manager-cainjector-559cdd8c87-bfxhc\" (UID: \"c135edcd-9eb2-4aaf-b63d-dd5505ce23f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" Jul 17 09:31:20.334152 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.333984 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c135edcd-9eb2-4aaf-b63d-dd5505ce23f4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bfxhc\" (UID: \"c135edcd-9eb2-4aaf-b63d-dd5505ce23f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" Jul 17 09:31:20.342158 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.342134 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c135edcd-9eb2-4aaf-b63d-dd5505ce23f4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bfxhc\" (UID: \"c135edcd-9eb2-4aaf-b63d-dd5505ce23f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" Jul 17 09:31:20.342306 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.342283 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4v9q\" (UniqueName: \"kubernetes.io/projected/c135edcd-9eb2-4aaf-b63d-dd5505ce23f4-kube-api-access-r4v9q\") pod \"cert-manager-cainjector-559cdd8c87-bfxhc\" (UID: \"c135edcd-9eb2-4aaf-b63d-dd5505ce23f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" Jul 17 09:31:20.410551 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.410516 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" Jul 17 09:31:20.562457 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:20.562416 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc"] Jul 17 09:31:20.612315 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:31:20.612231 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc135edcd_9eb2_4aaf_b63d_dd5505ce23f4.slice/crio-d33fc256208409c1ab794ef49c9a0f6dd0c55b54c12495b3d905805b8991608f WatchSource:0}: Error finding container d33fc256208409c1ab794ef49c9a0f6dd0c55b54c12495b3d905805b8991608f: Status 404 returned error can't find the container with id d33fc256208409c1ab794ef49c9a0f6dd0c55b54c12495b3d905805b8991608f Jul 17 09:31:21.226471 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:21.226440 2584 generic.go:356] "Generic (PLEG): container finished" podID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerID="03c8d9f24c78ac878a72c8503ca818180b79eec98750978e38a2dfc1e4d9eedc" exitCode=0 Jul 17 09:31:21.226872 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:21.226532 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" event={"ID":"c27dd28f-4d0d-46d1-8e0c-d698663db9c3","Type":"ContainerDied","Data":"03c8d9f24c78ac878a72c8503ca818180b79eec98750978e38a2dfc1e4d9eedc"} Jul 17 09:31:21.227734 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:21.227717 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" event={"ID":"c135edcd-9eb2-4aaf-b63d-dd5505ce23f4","Type":"ContainerStarted","Data":"d33fc256208409c1ab794ef49c9a0f6dd0c55b54c12495b3d905805b8991608f"} Jul 17 09:31:22.233064 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:22.233031 2584 generic.go:356] "Generic (PLEG): container finished" podID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerID="e1cb68001cfeef4be192c27c8f5cb2a60e63b1cb753b95eb5cb4d542a6b25b8d" exitCode=0 Jul 17 09:31:22.233524 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:22.233092 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" event={"ID":"c27dd28f-4d0d-46d1-8e0c-d698663db9c3","Type":"ContainerDied","Data":"e1cb68001cfeef4be192c27c8f5cb2a60e63b1cb753b95eb5cb4d542a6b25b8d"} Jul 17 09:31:23.357476 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.357452 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:23.460632 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.460605 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfb5g\" (UniqueName: \"kubernetes.io/projected/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-kube-api-access-wfb5g\") pod \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " Jul 17 09:31:23.460778 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.460706 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-util\") pod \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " Jul 17 09:31:23.460778 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.460734 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-bundle\") pod \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\" (UID: \"c27dd28f-4d0d-46d1-8e0c-d698663db9c3\") " Jul 17 09:31:23.461146 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.461111 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-bundle" (OuterVolumeSpecName: "bundle") pod "c27dd28f-4d0d-46d1-8e0c-d698663db9c3" (UID: "c27dd28f-4d0d-46d1-8e0c-d698663db9c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:31:23.462633 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.462612 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-kube-api-access-wfb5g" (OuterVolumeSpecName: "kube-api-access-wfb5g") pod "c27dd28f-4d0d-46d1-8e0c-d698663db9c3" (UID: "c27dd28f-4d0d-46d1-8e0c-d698663db9c3"). InnerVolumeSpecName "kube-api-access-wfb5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:31:23.465633 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.465611 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-util" (OuterVolumeSpecName: "util") pod "c27dd28f-4d0d-46d1-8e0c-d698663db9c3" (UID: "c27dd28f-4d0d-46d1-8e0c-d698663db9c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:31:23.561778 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.561709 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-util\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:23.561778 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.561745 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:23.561778 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:23.561757 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfb5g\" (UniqueName: \"kubernetes.io/projected/c27dd28f-4d0d-46d1-8e0c-d698663db9c3-kube-api-access-wfb5g\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:24.242469 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:24.242436 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" event={"ID":"c27dd28f-4d0d-46d1-8e0c-d698663db9c3","Type":"ContainerDied","Data":"60fcdfc0a47e68146fa70225463fe1b64d4a4824e09c3c1e12f6d488f4ad6f86"} Jul 17 09:31:24.242623 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:24.242476 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdnh6c" Jul 17 09:31:24.242623 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:24.242485 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60fcdfc0a47e68146fa70225463fe1b64d4a4824e09c3c1e12f6d488f4ad6f86" Jul 17 09:31:27.664312 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.664271 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h5cdj"] Jul 17 09:31:27.664768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.664631 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerName="pull" Jul 17 09:31:27.664768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.664647 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerName="pull" Jul 17 09:31:27.664768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.664673 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerName="util" Jul 17 09:31:27.664768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.664680 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerName="util" Jul 17 09:31:27.664768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.664689 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerName="extract" Jul 17 09:31:27.664768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.664694 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerName="extract" Jul 17 09:31:27.664768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.664752 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27dd28f-4d0d-46d1-8e0c-d698663db9c3" containerName="extract" Jul 17 09:31:27.666938 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.666920 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-h5cdj" Jul 17 09:31:27.669693 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.669672 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jdcc7\"" Jul 17 09:31:27.678515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.678493 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h5cdj"] Jul 17 09:31:27.799308 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.799273 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0711e19-f461-46d1-af08-72a9d65ceeb6-bound-sa-token\") pod \"cert-manager-55b4d954d5-h5cdj\" (UID: \"a0711e19-f461-46d1-af08-72a9d65ceeb6\") " pod="cert-manager/cert-manager-55b4d954d5-h5cdj" Jul 17 09:31:27.799523 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.799340 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxprx\" (UniqueName: \"kubernetes.io/projected/a0711e19-f461-46d1-af08-72a9d65ceeb6-kube-api-access-zxprx\") pod \"cert-manager-55b4d954d5-h5cdj\" (UID: \"a0711e19-f461-46d1-af08-72a9d65ceeb6\") " pod="cert-manager/cert-manager-55b4d954d5-h5cdj" Jul 17 09:31:27.900764 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.900726 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxprx\" (UniqueName: \"kubernetes.io/projected/a0711e19-f461-46d1-af08-72a9d65ceeb6-kube-api-access-zxprx\") pod \"cert-manager-55b4d954d5-h5cdj\" (UID: \"a0711e19-f461-46d1-af08-72a9d65ceeb6\") " pod="cert-manager/cert-manager-55b4d954d5-h5cdj" Jul 17 09:31:27.900930 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.900818 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0711e19-f461-46d1-af08-72a9d65ceeb6-bound-sa-token\") pod \"cert-manager-55b4d954d5-h5cdj\" (UID: \"a0711e19-f461-46d1-af08-72a9d65ceeb6\") " pod="cert-manager/cert-manager-55b4d954d5-h5cdj" Jul 17 09:31:27.909750 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.909721 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxprx\" (UniqueName: \"kubernetes.io/projected/a0711e19-f461-46d1-af08-72a9d65ceeb6-kube-api-access-zxprx\") pod \"cert-manager-55b4d954d5-h5cdj\" (UID: \"a0711e19-f461-46d1-af08-72a9d65ceeb6\") " pod="cert-manager/cert-manager-55b4d954d5-h5cdj" Jul 17 09:31:27.910120 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.910096 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a0711e19-f461-46d1-af08-72a9d65ceeb6-bound-sa-token\") pod \"cert-manager-55b4d954d5-h5cdj\" (UID: \"a0711e19-f461-46d1-af08-72a9d65ceeb6\") " pod="cert-manager/cert-manager-55b4d954d5-h5cdj" Jul 17 09:31:27.978994 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:27.978922 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-h5cdj" Jul 17 09:31:28.970205 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:28.970131 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-h5cdj"] Jul 17 09:31:28.972428 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:31:28.972377 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0711e19_f461_46d1_af08_72a9d65ceeb6.slice/crio-137ac69defa429b936b71960ef9e5e4564d7cb37ada2bd976ccb68013ab9510d WatchSource:0}: Error finding container 137ac69defa429b936b71960ef9e5e4564d7cb37ada2bd976ccb68013ab9510d: Status 404 returned error can't find the container with id 137ac69defa429b936b71960ef9e5e4564d7cb37ada2bd976ccb68013ab9510d Jul 17 09:31:29.266675 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:29.266574 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" event={"ID":"c135edcd-9eb2-4aaf-b63d-dd5505ce23f4","Type":"ContainerStarted","Data":"3c8b757a86733b8bc1e9d935b03b3add010d89bcafe70c6491ce1f4ee9735610"} Jul 17 09:31:29.268115 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:29.268081 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-h5cdj" event={"ID":"a0711e19-f461-46d1-af08-72a9d65ceeb6","Type":"ContainerStarted","Data":"5899fa8cdbc6d4333cb42a5f1de556d5d2bbf46731bbad8668a05314f7d52c4c"} Jul 17 09:31:29.268115 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:29.268114 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-h5cdj" event={"ID":"a0711e19-f461-46d1-af08-72a9d65ceeb6","Type":"ContainerStarted","Data":"137ac69defa429b936b71960ef9e5e4564d7cb37ada2bd976ccb68013ab9510d"} Jul 17 09:31:29.281931 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:29.281885 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bfxhc" podStartSLOduration=1.016308413 podStartE2EDuration="9.281870683s" totalImagesPullingTime="8.26556227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:31:29.28082183 +0000 UTC m=+271.820020797" watchObservedRunningTime="2026-07-17 09:31:29.281870683 +0000 UTC m=+271.821069648" Jul 17 09:31:29.297979 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:29.297932 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-h5cdj" podStartSLOduration=2.29791919 podStartE2EDuration="2.29791919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:31:29.296490283 +0000 UTC m=+271.835689262" watchObservedRunningTime="2026-07-17 09:31:29.29791919 +0000 UTC m=+271.837118155" Jul 17 09:31:36.659973 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.659931 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7"] Jul 17 09:31:36.662512 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.662490 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.664997 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.664978 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 17 09:31:36.665094 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.665077 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-79wp7\"" Jul 17 09:31:36.665966 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.665938 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 17 09:31:36.671413 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.671367 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7"] Jul 17 09:31:36.774110 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.774079 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.774257 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.774142 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8h2t\" (UniqueName: \"kubernetes.io/projected/ac317528-cd29-4787-9b20-81b120d0b52f-kube-api-access-c8h2t\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.774257 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.774225 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.875264 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.875231 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.875454 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.875280 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8h2t\" (UniqueName: \"kubernetes.io/projected/ac317528-cd29-4787-9b20-81b120d0b52f-kube-api-access-c8h2t\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.875454 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.875317 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.875633 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.875615 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.875702 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.875647 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.883352 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.883319 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8h2t\" (UniqueName: \"kubernetes.io/projected/ac317528-cd29-4787-9b20-81b120d0b52f-kube-api-access-c8h2t\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:36.971786 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:36.971720 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:37.094735 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:37.094712 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7"] Jul 17 09:31:37.096594 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:31:37.096566 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac317528_cd29_4787_9b20_81b120d0b52f.slice/crio-9b9e36db26f76912656fd21f9f6c6e214eaf54f2c6436816ed01b963a4722f1a WatchSource:0}: Error finding container 9b9e36db26f76912656fd21f9f6c6e214eaf54f2c6436816ed01b963a4722f1a: Status 404 returned error can't find the container with id 9b9e36db26f76912656fd21f9f6c6e214eaf54f2c6436816ed01b963a4722f1a Jul 17 09:31:37.305840 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:37.305758 2584 generic.go:356] "Generic (PLEG): container finished" podID="ac317528-cd29-4787-9b20-81b120d0b52f" containerID="4d33b34c3f712132652b6fb7f2e76e0f1b7e526d6fa17f13b18df6275b0fdea6" exitCode=0 Jul 17 09:31:37.305980 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:37.305847 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" event={"ID":"ac317528-cd29-4787-9b20-81b120d0b52f","Type":"ContainerDied","Data":"4d33b34c3f712132652b6fb7f2e76e0f1b7e526d6fa17f13b18df6275b0fdea6"} Jul 17 09:31:37.305980 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:37.305895 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" event={"ID":"ac317528-cd29-4787-9b20-81b120d0b52f","Type":"ContainerStarted","Data":"9b9e36db26f76912656fd21f9f6c6e214eaf54f2c6436816ed01b963a4722f1a"} Jul 17 09:31:38.311675 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:38.311593 2584 generic.go:356] "Generic (PLEG): container finished" podID="ac317528-cd29-4787-9b20-81b120d0b52f" containerID="9d2976586fb9794d8327ab0c5d0fc9c1876a12c98db852521aceddad4c4d38be" exitCode=0 Jul 17 09:31:38.312024 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:38.311672 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" event={"ID":"ac317528-cd29-4787-9b20-81b120d0b52f","Type":"ContainerDied","Data":"9d2976586fb9794d8327ab0c5d0fc9c1876a12c98db852521aceddad4c4d38be"} Jul 17 09:31:39.317899 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:39.317867 2584 generic.go:356] "Generic (PLEG): container finished" podID="ac317528-cd29-4787-9b20-81b120d0b52f" containerID="b77a00d6b62450e2cb3c91d17619463501f9733d511e9e8f3497503aa1f8b0db" exitCode=0 Jul 17 09:31:39.318359 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:39.317929 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" event={"ID":"ac317528-cd29-4787-9b20-81b120d0b52f","Type":"ContainerDied","Data":"b77a00d6b62450e2cb3c91d17619463501f9733d511e9e8f3497503aa1f8b0db"} Jul 17 09:31:40.444668 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.444645 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:40.505837 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.505813 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-bundle\") pod \"ac317528-cd29-4787-9b20-81b120d0b52f\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " Jul 17 09:31:40.505971 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.505848 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8h2t\" (UniqueName: \"kubernetes.io/projected/ac317528-cd29-4787-9b20-81b120d0b52f-kube-api-access-c8h2t\") pod \"ac317528-cd29-4787-9b20-81b120d0b52f\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " Jul 17 09:31:40.505971 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.505876 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-util\") pod \"ac317528-cd29-4787-9b20-81b120d0b52f\" (UID: \"ac317528-cd29-4787-9b20-81b120d0b52f\") " Jul 17 09:31:40.506624 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.506595 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-bundle" (OuterVolumeSpecName: "bundle") pod "ac317528-cd29-4787-9b20-81b120d0b52f" (UID: "ac317528-cd29-4787-9b20-81b120d0b52f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:31:40.507878 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.507845 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac317528-cd29-4787-9b20-81b120d0b52f-kube-api-access-c8h2t" (OuterVolumeSpecName: "kube-api-access-c8h2t") pod "ac317528-cd29-4787-9b20-81b120d0b52f" (UID: "ac317528-cd29-4787-9b20-81b120d0b52f"). InnerVolumeSpecName "kube-api-access-c8h2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:31:40.511680 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.511644 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-util" (OuterVolumeSpecName: "util") pod "ac317528-cd29-4787-9b20-81b120d0b52f" (UID: "ac317528-cd29-4787-9b20-81b120d0b52f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:31:40.606997 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.606972 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:40.607096 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.606996 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8h2t\" (UniqueName: \"kubernetes.io/projected/ac317528-cd29-4787-9b20-81b120d0b52f-kube-api-access-c8h2t\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:40.607096 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:40.607010 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac317528-cd29-4787-9b20-81b120d0b52f-util\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:31:41.328314 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:41.328282 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" Jul 17 09:31:41.328314 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:41.328293 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp5pn7" event={"ID":"ac317528-cd29-4787-9b20-81b120d0b52f","Type":"ContainerDied","Data":"9b9e36db26f76912656fd21f9f6c6e214eaf54f2c6436816ed01b963a4722f1a"} Jul 17 09:31:41.328533 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:41.328338 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9e36db26f76912656fd21f9f6c6e214eaf54f2c6436816ed01b963a4722f1a" Jul 17 09:31:51.100524 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.100492 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw"] Jul 17 09:31:51.100892 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.100811 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac317528-cd29-4787-9b20-81b120d0b52f" containerName="util" Jul 17 09:31:51.100892 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.100821 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac317528-cd29-4787-9b20-81b120d0b52f" containerName="util" Jul 17 09:31:51.100892 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.100829 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac317528-cd29-4787-9b20-81b120d0b52f" containerName="extract" Jul 17 09:31:51.100892 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.100834 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac317528-cd29-4787-9b20-81b120d0b52f" containerName="extract" Jul 17 09:31:51.100892 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.100858 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac317528-cd29-4787-9b20-81b120d0b52f" containerName="pull" Jul 17 09:31:51.100892 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.100864 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac317528-cd29-4787-9b20-81b120d0b52f" containerName="pull" Jul 17 09:31:51.101072 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.100911 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac317528-cd29-4787-9b20-81b120d0b52f" containerName="extract" Jul 17 09:31:51.108075 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.108054 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.112437 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.112414 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 09:31:51.112570 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.112518 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 09:31:51.112655 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.112633 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 09:31:51.112775 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.112518 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 09:31:51.112775 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.112742 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hcnr9\"" Jul 17 09:31:51.112875 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.112799 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 09:31:51.114443 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.114420 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw"] Jul 17 09:31:51.177413 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.177370 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6672feea-572a-4101-8b3c-027c97014e5c-cert\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.177546 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.177420 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6672feea-572a-4101-8b3c-027c97014e5c-metrics-cert\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.177546 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.177453 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6672feea-572a-4101-8b3c-027c97014e5c-manager-config\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.177546 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.177512 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546mz\" (UniqueName: \"kubernetes.io/projected/6672feea-572a-4101-8b3c-027c97014e5c-kube-api-access-546mz\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.278380 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.278345 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6672feea-572a-4101-8b3c-027c97014e5c-cert\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.278380 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.278383 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6672feea-572a-4101-8b3c-027c97014e5c-metrics-cert\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.278599 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.278443 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6672feea-572a-4101-8b3c-027c97014e5c-manager-config\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.278599 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.278469 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546mz\" (UniqueName: \"kubernetes.io/projected/6672feea-572a-4101-8b3c-027c97014e5c-kube-api-access-546mz\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.279267 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.279237 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6672feea-572a-4101-8b3c-027c97014e5c-manager-config\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.281015 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.280988 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6672feea-572a-4101-8b3c-027c97014e5c-cert\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.281117 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.281022 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6672feea-572a-4101-8b3c-027c97014e5c-metrics-cert\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.291188 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.291167 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-546mz\" (UniqueName: \"kubernetes.io/projected/6672feea-572a-4101-8b3c-027c97014e5c-kube-api-access-546mz\") pod \"lws-controller-manager-7df87f8d9d-hl2cw\" (UID: \"6672feea-572a-4101-8b3c-027c97014e5c\") " pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.418108 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.418038 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:51.517225 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.517192 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd"] Jul 17 09:31:51.521113 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.521086 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.524184 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.524156 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7g64w\"" Jul 17 09:31:51.525285 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.524522 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 09:31:51.525285 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.524681 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 09:31:51.525285 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.524739 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 09:31:51.525285 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.524790 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 09:31:51.538253 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.538224 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd"] Jul 17 09:31:51.571438 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.571418 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw"] Jul 17 09:31:51.573876 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:31:51.573851 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6672feea_572a_4101_8b3c_027c97014e5c.slice/crio-118ffb5b286ad4358f62ae2b0e1bc18e6158f2efafc884292f3390317ec8d902 WatchSource:0}: Error finding container 118ffb5b286ad4358f62ae2b0e1bc18e6158f2efafc884292f3390317ec8d902: Status 404 returned error can't find the container with id 118ffb5b286ad4358f62ae2b0e1bc18e6158f2efafc884292f3390317ec8d902 Jul 17 09:31:51.580956 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.580934 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce9315d2-e289-412c-b70b-3fb424690668-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.581042 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.580990 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6r5\" (UniqueName: \"kubernetes.io/projected/ce9315d2-e289-412c-b70b-3fb424690668-kube-api-access-vd6r5\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.581099 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.581048 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce9315d2-e289-412c-b70b-3fb424690668-manifests\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.581173 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.581115 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9315d2-e289-412c-b70b-3fb424690668-tmp\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.581173 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.581141 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce9315d2-e289-412c-b70b-3fb424690668-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.682160 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.682095 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce9315d2-e289-412c-b70b-3fb424690668-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.682160 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.682142 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6r5\" (UniqueName: \"kubernetes.io/projected/ce9315d2-e289-412c-b70b-3fb424690668-kube-api-access-vd6r5\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.682305 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.682181 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce9315d2-e289-412c-b70b-3fb424690668-manifests\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.682305 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.682233 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9315d2-e289-412c-b70b-3fb424690668-tmp\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.682305 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.682254 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce9315d2-e289-412c-b70b-3fb424690668-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.682705 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.682676 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce9315d2-e289-412c-b70b-3fb424690668-manifests\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.684486 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.684468 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9315d2-e289-412c-b70b-3fb424690668-tmp\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.684727 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.684704 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce9315d2-e289-412c-b70b-3fb424690668-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.684779 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.684754 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce9315d2-e289-412c-b70b-3fb424690668-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.702467 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.702443 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6r5\" (UniqueName: \"kubernetes.io/projected/ce9315d2-e289-412c-b70b-3fb424690668-kube-api-access-vd6r5\") pod \"opendatahub-operator-controller-manager-6b5566fdf9-4ncqd\" (UID: \"ce9315d2-e289-412c-b70b-3fb424690668\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.834960 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.834926 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:51.971583 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:51.971560 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd"] Jul 17 09:31:51.973204 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:31:51.973177 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9315d2_e289_412c_b70b_3fb424690668.slice/crio-5de300558932b1b3668fc1ce83c83585bd815270bbee535d4340bf04d33f34e0 WatchSource:0}: Error finding container 5de300558932b1b3668fc1ce83c83585bd815270bbee535d4340bf04d33f34e0: Status 404 returned error can't find the container with id 5de300558932b1b3668fc1ce83c83585bd815270bbee535d4340bf04d33f34e0 Jul 17 09:31:52.376271 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:52.376237 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" event={"ID":"ce9315d2-e289-412c-b70b-3fb424690668","Type":"ContainerStarted","Data":"5de300558932b1b3668fc1ce83c83585bd815270bbee535d4340bf04d33f34e0"} Jul 17 09:31:52.377471 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:52.377441 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" event={"ID":"6672feea-572a-4101-8b3c-027c97014e5c","Type":"ContainerStarted","Data":"118ffb5b286ad4358f62ae2b0e1bc18e6158f2efafc884292f3390317ec8d902"} Jul 17 09:31:53.385710 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:53.385671 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" event={"ID":"6672feea-572a-4101-8b3c-027c97014e5c","Type":"ContainerStarted","Data":"985bd06e8d1b6eab4b09aafc16a78b0dc6d175678a5f07c51d4a86970efe1e36"} Jul 17 09:31:53.386164 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:53.385799 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:31:53.402720 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:53.402631 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" podStartSLOduration=0.94642888 podStartE2EDuration="2.402612237s" totalImagesPullingTime="1.456183357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:31:53.401218084 +0000 UTC m=+295.940417051" watchObservedRunningTime="2026-07-17 09:31:53.402612237 +0000 UTC m=+295.941811203" Jul 17 09:31:55.399423 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:55.399368 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" event={"ID":"ce9315d2-e289-412c-b70b-3fb424690668","Type":"ContainerStarted","Data":"fabce8a52a47ba36d155b2b03631713100338b6cdeed3b18906bc43c5757974e"} Jul 17 09:31:56.405062 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:56.405027 2584 generic.go:356] "Generic (PLEG): container finished" podID="ce9315d2-e289-412c-b70b-3fb424690668" containerID="fabce8a52a47ba36d155b2b03631713100338b6cdeed3b18906bc43c5757974e" exitCode=0 Jul 17 09:31:56.405447 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:56.405089 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" event={"ID":"ce9315d2-e289-412c-b70b-3fb424690668","Type":"ContainerDied","Data":"fabce8a52a47ba36d155b2b03631713100338b6cdeed3b18906bc43c5757974e"} Jul 17 09:31:57.411699 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:57.411659 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" event={"ID":"ce9315d2-e289-412c-b70b-3fb424690668","Type":"ContainerStarted","Data":"1967130df8cc0fff2db4097a464d8d203956802e41761d8a0d02e65d3453cdda"} Jul 17 09:31:57.412085 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:57.411799 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:31:57.433993 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:57.433946 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" podStartSLOduration=2.062782406 podStartE2EDuration="6.433932252s" totalImagesPullingTime="3.349486023s" totalInitContainerRuntime="1.021663823s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:31:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:31:57.431366403 +0000 UTC m=+299.970565384" watchObservedRunningTime="2026-07-17 09:31:57.433932252 +0000 UTC m=+299.973131217" Jul 17 09:31:57.860283 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:57.860256 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:31:57.868776 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:57.868716 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:31:57.871475 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:31:57.871454 2584 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 09:32:04.393309 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:04.393278 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7df87f8d9d-hl2cw" Jul 17 09:32:08.418328 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:08.418297 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b5566fdf9-4ncqd" Jul 17 09:32:19.828251 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.828220 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g"] Jul 17 09:32:19.830620 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.830602 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.833135 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.833112 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 09:32:19.833286 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.833148 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 09:32:19.833286 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.833224 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 09:32:19.833286 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.833253 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lnvpk\"" Jul 17 09:32:19.843698 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.843676 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g"] Jul 17 09:32:19.912705 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912673 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.912836 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912711 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.912836 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912754 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.912836 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912783 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.912836 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912806 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.912836 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912835 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.913016 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912869 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.913016 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912902 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b2d768-3b69-4816-9ac3-1ec68ded2216-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:19.913016 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:19.912943 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpvt\" (UniqueName: \"kubernetes.io/projected/82b2d768-3b69-4816-9ac3-1ec68ded2216-kube-api-access-qkpvt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013361 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013331 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpvt\" (UniqueName: \"kubernetes.io/projected/82b2d768-3b69-4816-9ac3-1ec68ded2216-kube-api-access-qkpvt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013529 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013385 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013529 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013448 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013529 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013479 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013529 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013508 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013713 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013535 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013713 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013566 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013713 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013595 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013713 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013624 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b2d768-3b69-4816-9ac3-1ec68ded2216-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.013920 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.013844 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.014047 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.014011 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.014136 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.014072 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.014190 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.014137 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.014287 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.014270 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b2d768-3b69-4816-9ac3-1ec68ded2216-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.015724 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.015702 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.015974 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.015958 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.020869 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.020850 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpvt\" (UniqueName: \"kubernetes.io/projected/82b2d768-3b69-4816-9ac3-1ec68ded2216-kube-api-access-qkpvt\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.021026 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.021010 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b2d768-3b69-4816-9ac3-1ec68ded2216-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4ngb4g\" (UID: \"82b2d768-3b69-4816-9ac3-1ec68ded2216\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.140080 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.140051 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:20.272226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.272204 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g"] Jul 17 09:32:20.274304 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:32:20.274271 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b2d768_3b69_4816_9ac3_1ec68ded2216.slice/crio-c334364f2fbb3cc82069fb1f11b0db2bc84ea02a535215ec9c748e58720a215c WatchSource:0}: Error finding container c334364f2fbb3cc82069fb1f11b0db2bc84ea02a535215ec9c748e58720a215c: Status 404 returned error can't find the container with id c334364f2fbb3cc82069fb1f11b0db2bc84ea02a535215ec9c748e58720a215c Jul 17 09:32:20.276376 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.276295 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 09:32:20.523501 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:20.523422 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" event={"ID":"82b2d768-3b69-4816-9ac3-1ec68ded2216","Type":"ContainerStarted","Data":"c334364f2fbb3cc82069fb1f11b0db2bc84ea02a535215ec9c748e58720a215c"} Jul 17 09:32:21.912191 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:21.912156 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966"] Jul 17 09:32:21.918490 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:21.918457 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:21.922026 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:21.921828 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 09:32:21.922026 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:21.921828 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 09:32:21.922208 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:21.922190 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 09:32:21.923109 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:21.923091 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pxfmw\"" Jul 17 09:32:21.955605 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:21.955579 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966"] Jul 17 09:32:22.029414 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.029369 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.029576 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.029463 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.029576 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.029502 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.029576 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.029530 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.029713 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.029605 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbc9h\" (UniqueName: \"kubernetes.io/projected/a6e951b2-4383-4adf-9e5d-0291652188a3-kube-api-access-hbc9h\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.029713 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.029627 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e951b2-4383-4adf-9e5d-0291652188a3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.029713 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.029661 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.130471 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.130438 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.130630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.130484 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.130630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.130516 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.130630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.130550 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.130630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.130596 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbc9h\" (UniqueName: \"kubernetes.io/projected/a6e951b2-4383-4adf-9e5d-0291652188a3-kube-api-access-hbc9h\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.130630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.130621 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e951b2-4383-4adf-9e5d-0291652188a3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.130889 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.130652 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.131462 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.131436 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.133307 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.133282 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.133548 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.133514 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e951b2-4383-4adf-9e5d-0291652188a3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.133661 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.133640 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.133771 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.133742 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.139436 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.139417 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbc9h\" (UniqueName: \"kubernetes.io/projected/a6e951b2-4383-4adf-9e5d-0291652188a3-kube-api-access-hbc9h\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.139726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.139707 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e951b2-4383-4adf-9e5d-0291652188a3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-m8966\" (UID: \"a6e951b2-4383-4adf-9e5d-0291652188a3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.230481 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.230384 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:22.630294 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.630235 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:32:22.630434 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.630321 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:32:22.630434 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.630368 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:32:22.740521 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:22.740493 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966"] Jul 17 09:32:22.743472 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:32:22.743447 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e951b2_4383_4adf_9e5d_0291652188a3.slice/crio-8959503ef9a8e17924aebc2197c2cdaac312b878f6f8b2b62399da743edb3a15 WatchSource:0}: Error finding container 8959503ef9a8e17924aebc2197c2cdaac312b878f6f8b2b62399da743edb3a15: Status 404 returned error can't find the container with id 8959503ef9a8e17924aebc2197c2cdaac312b878f6f8b2b62399da743edb3a15 Jul 17 09:32:23.540671 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:23.540636 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" event={"ID":"a6e951b2-4383-4adf-9e5d-0291652188a3","Type":"ContainerStarted","Data":"8959503ef9a8e17924aebc2197c2cdaac312b878f6f8b2b62399da743edb3a15"} Jul 17 09:32:23.542460 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:23.542433 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" event={"ID":"82b2d768-3b69-4816-9ac3-1ec68ded2216","Type":"ContainerStarted","Data":"fc24ae0cc6bfe645a2e23f2c58fd11ba041d7a24d819190d40431af279ca5edc"} Jul 17 09:32:24.140348 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:24.140311 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:24.162349 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:24.162310 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:24.198536 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:24.198470 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" podStartSLOduration=2.845030258 podStartE2EDuration="5.198449214s" totalImagesPullingTime="2.353418956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:32:23.597565392 +0000 UTC m=+326.136764359" watchObservedRunningTime="2026-07-17 09:32:24.198449214 +0000 UTC m=+326.737648179" Jul 17 09:32:24.548366 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:24.548293 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:24.549512 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:24.549490 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4ngb4g" Jul 17 09:32:25.091183 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:25.091152 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:32:25.091275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:25.091213 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:32:25.554617 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:25.554518 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" event={"ID":"a6e951b2-4383-4adf-9e5d-0291652188a3","Type":"ContainerStarted","Data":"c6c6979c25b201117f3e01c367502bbfccb4cb2be56992d2526441e9282756a6"} Jul 17 09:32:25.589608 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:25.589555 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" podStartSLOduration=2.244191155 podStartE2EDuration="4.589534764s" totalImagesPullingTime="2.345343609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:32:25.58952648 +0000 UTC m=+328.128725448" watchObservedRunningTime="2026-07-17 09:32:25.589534764 +0000 UTC m=+328.128733726" Jul 17 09:32:26.558904 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:26.558871 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:32:26.560224 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:32:26.560201 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-m8966" Jul 17 09:33:28.879318 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:28.879278 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26"] Jul 17 09:33:28.881867 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:28.881847 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:28.884549 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:28.884528 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 09:33:28.884643 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:28.884569 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 09:33:28.885755 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:28.885730 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k2wpb\"" Jul 17 09:33:28.890797 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:28.890762 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26"] Jul 17 09:33:29.009642 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.009602 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.009642 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.009643 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.009852 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.009706 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8r8l\" (UniqueName: \"kubernetes.io/projected/a26f9c86-140d-454d-9c10-9acd72107ac0-kube-api-access-s8r8l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.111117 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.111077 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.111117 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.111116 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.111295 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.111152 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8r8l\" (UniqueName: \"kubernetes.io/projected/a26f9c86-140d-454d-9c10-9acd72107ac0-kube-api-access-s8r8l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.111487 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.111466 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.111546 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.111524 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.120065 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.120043 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8r8l\" (UniqueName: \"kubernetes.io/projected/a26f9c86-140d-454d-9c10-9acd72107ac0-kube-api-access-s8r8l\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.192329 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.192251 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:29.328451 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.328426 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26"] Jul 17 09:33:29.330528 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:33:29.330501 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26f9c86_140d_454d_9c10_9acd72107ac0.slice/crio-1d9f9836ba21bca460b33a743deaea9e2ceea40db7e64884231d86a03429144e WatchSource:0}: Error finding container 1d9f9836ba21bca460b33a743deaea9e2ceea40db7e64884231d86a03429144e: Status 404 returned error can't find the container with id 1d9f9836ba21bca460b33a743deaea9e2ceea40db7e64884231d86a03429144e Jul 17 09:33:29.680510 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.680476 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd"] Jul 17 09:33:29.683134 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.683117 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.692105 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.692083 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd"] Jul 17 09:33:29.818033 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.817992 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.818033 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.818031 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.818251 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.818060 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h82p6\" (UniqueName: \"kubernetes.io/projected/32491371-c26f-4d72-ab8a-308769d06bcf-kube-api-access-h82p6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.856820 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.856786 2584 generic.go:356] "Generic (PLEG): container finished" podID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerID="60c9a3217adb7d4b153aac82845b69746b2189835af3d21ece46f94d5bcc839a" exitCode=0 Jul 17 09:33:29.857010 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.856874 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" event={"ID":"a26f9c86-140d-454d-9c10-9acd72107ac0","Type":"ContainerDied","Data":"60c9a3217adb7d4b153aac82845b69746b2189835af3d21ece46f94d5bcc839a"} Jul 17 09:33:29.857010 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.856923 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" event={"ID":"a26f9c86-140d-454d-9c10-9acd72107ac0","Type":"ContainerStarted","Data":"1d9f9836ba21bca460b33a743deaea9e2ceea40db7e64884231d86a03429144e"} Jul 17 09:33:29.919321 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.919285 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.919819 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.919331 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.919819 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.919367 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h82p6\" (UniqueName: \"kubernetes.io/projected/32491371-c26f-4d72-ab8a-308769d06bcf-kube-api-access-h82p6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.919819 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.919712 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.919819 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.919735 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.928094 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.928065 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h82p6\" (UniqueName: \"kubernetes.io/projected/32491371-c26f-4d72-ab8a-308769d06bcf-kube-api-access-h82p6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:29.993102 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:29.993023 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:30.120585 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.120549 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd"] Jul 17 09:33:30.122296 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:33:30.122270 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32491371_c26f_4d72_ab8a_308769d06bcf.slice/crio-3d59427a1db1681e467379b6b774b2b69a51e12e017b789c4e881cb7d9cd5897 WatchSource:0}: Error finding container 3d59427a1db1681e467379b6b774b2b69a51e12e017b789c4e881cb7d9cd5897: Status 404 returned error can't find the container with id 3d59427a1db1681e467379b6b774b2b69a51e12e017b789c4e881cb7d9cd5897 Jul 17 09:33:30.276261 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.276182 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j"] Jul 17 09:33:30.278680 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.278661 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.286445 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.286417 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j"] Jul 17 09:33:30.423824 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.423793 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.423995 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.423855 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpkk\" (UniqueName: \"kubernetes.io/projected/8257a986-e99b-4089-8b45-2b41e0340103-kube-api-access-qgpkk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.423995 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.423948 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.525294 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.525264 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpkk\" (UniqueName: \"kubernetes.io/projected/8257a986-e99b-4089-8b45-2b41e0340103-kube-api-access-qgpkk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.525463 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.525326 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.525463 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.525387 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.525756 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.525733 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.525823 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.525754 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.532848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.532787 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpkk\" (UniqueName: \"kubernetes.io/projected/8257a986-e99b-4089-8b45-2b41e0340103-kube-api-access-qgpkk\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.589623 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.589589 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:30.677467 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.677433 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259"] Jul 17 09:33:30.681025 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.681004 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.689634 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.689581 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259"] Jul 17 09:33:30.746498 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.746473 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j"] Jul 17 09:33:30.748235 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:33:30.748209 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8257a986_e99b_4089_8b45_2b41e0340103.slice/crio-08a6641593d522655d8003c57fbe96f2d2adc50335eb05168d0183ebcbdd8646 WatchSource:0}: Error finding container 08a6641593d522655d8003c57fbe96f2d2adc50335eb05168d0183ebcbdd8646: Status 404 returned error can't find the container with id 08a6641593d522655d8003c57fbe96f2d2adc50335eb05168d0183ebcbdd8646 Jul 17 09:33:30.829234 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.829195 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.829428 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.829243 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ft9b\" (UniqueName: \"kubernetes.io/projected/937498cf-9879-45b0-92ca-3c106854f506-kube-api-access-5ft9b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.829428 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.829269 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.864329 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.864298 2584 generic.go:356] "Generic (PLEG): container finished" podID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerID="174338adb7f028827fb82ce893b196fb0423d1513f8327d90a8f0cb8b43a2929" exitCode=0 Jul 17 09:33:30.864522 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.864391 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" event={"ID":"a26f9c86-140d-454d-9c10-9acd72107ac0","Type":"ContainerDied","Data":"174338adb7f028827fb82ce893b196fb0423d1513f8327d90a8f0cb8b43a2929"} Jul 17 09:33:30.866147 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.866126 2584 generic.go:356] "Generic (PLEG): container finished" podID="32491371-c26f-4d72-ab8a-308769d06bcf" containerID="945c9d2b471e5aa9731b62cd328002b6882662b3b9247b109552d3208d95531b" exitCode=0 Jul 17 09:33:30.866247 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.866199 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" event={"ID":"32491371-c26f-4d72-ab8a-308769d06bcf","Type":"ContainerDied","Data":"945c9d2b471e5aa9731b62cd328002b6882662b3b9247b109552d3208d95531b"} Jul 17 09:33:30.866247 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.866238 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" event={"ID":"32491371-c26f-4d72-ab8a-308769d06bcf","Type":"ContainerStarted","Data":"3d59427a1db1681e467379b6b774b2b69a51e12e017b789c4e881cb7d9cd5897"} Jul 17 09:33:30.873867 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.870971 2584 generic.go:356] "Generic (PLEG): container finished" podID="8257a986-e99b-4089-8b45-2b41e0340103" containerID="2657b07bf7d96c70f1a13d174e2e4e0e090e22986ee834b0d318ef0351165012" exitCode=0 Jul 17 09:33:30.873867 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.871077 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" event={"ID":"8257a986-e99b-4089-8b45-2b41e0340103","Type":"ContainerDied","Data":"2657b07bf7d96c70f1a13d174e2e4e0e090e22986ee834b0d318ef0351165012"} Jul 17 09:33:30.873867 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.871149 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" event={"ID":"8257a986-e99b-4089-8b45-2b41e0340103","Type":"ContainerStarted","Data":"08a6641593d522655d8003c57fbe96f2d2adc50335eb05168d0183ebcbdd8646"} Jul 17 09:33:30.929878 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.929846 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ft9b\" (UniqueName: \"kubernetes.io/projected/937498cf-9879-45b0-92ca-3c106854f506-kube-api-access-5ft9b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.930179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.929885 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.930179 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.930031 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.930253 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.930237 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.930318 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.930302 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.937151 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.937126 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ft9b\" (UniqueName: \"kubernetes.io/projected/937498cf-9879-45b0-92ca-3c106854f506-kube-api-access-5ft9b\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:30.990884 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:30.990850 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:31.123712 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.123685 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259"] Jul 17 09:33:31.126023 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:33:31.125983 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937498cf_9879_45b0_92ca_3c106854f506.slice/crio-f6cd3ef3ce1c86df7346976fe9b6cb9baefc7c9f9c69dec73105e11c0fb6657d WatchSource:0}: Error finding container f6cd3ef3ce1c86df7346976fe9b6cb9baefc7c9f9c69dec73105e11c0fb6657d: Status 404 returned error can't find the container with id f6cd3ef3ce1c86df7346976fe9b6cb9baefc7c9f9c69dec73105e11c0fb6657d Jul 17 09:33:31.878242 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.878208 2584 generic.go:356] "Generic (PLEG): container finished" podID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerID="c7f301586332715248c0f041ce92ad43612a584a17980c6486291b1f6ab5d0a8" exitCode=0 Jul 17 09:33:31.878375 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.878284 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" event={"ID":"a26f9c86-140d-454d-9c10-9acd72107ac0","Type":"ContainerDied","Data":"c7f301586332715248c0f041ce92ad43612a584a17980c6486291b1f6ab5d0a8"} Jul 17 09:33:31.879560 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.879541 2584 generic.go:356] "Generic (PLEG): container finished" podID="937498cf-9879-45b0-92ca-3c106854f506" containerID="de81377b6ab883ee4a1df00dcdc01e04cea45bb67c31d2780e25c3cdda08d29f" exitCode=0 Jul 17 09:33:31.879632 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.879612 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" event={"ID":"937498cf-9879-45b0-92ca-3c106854f506","Type":"ContainerDied","Data":"de81377b6ab883ee4a1df00dcdc01e04cea45bb67c31d2780e25c3cdda08d29f"} Jul 17 09:33:31.879688 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.879650 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" event={"ID":"937498cf-9879-45b0-92ca-3c106854f506","Type":"ContainerStarted","Data":"f6cd3ef3ce1c86df7346976fe9b6cb9baefc7c9f9c69dec73105e11c0fb6657d"} Jul 17 09:33:31.881339 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.881320 2584 generic.go:356] "Generic (PLEG): container finished" podID="32491371-c26f-4d72-ab8a-308769d06bcf" containerID="744901d3eb62402ab665bb9f8dae3637a857dc66f0b9ac63d445fd4b0ed04d80" exitCode=0 Jul 17 09:33:31.881458 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.881440 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" event={"ID":"32491371-c26f-4d72-ab8a-308769d06bcf","Type":"ContainerDied","Data":"744901d3eb62402ab665bb9f8dae3637a857dc66f0b9ac63d445fd4b0ed04d80"} Jul 17 09:33:31.883245 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.883227 2584 generic.go:356] "Generic (PLEG): container finished" podID="8257a986-e99b-4089-8b45-2b41e0340103" containerID="139169ce7e1032522e3cbe5f0d24257b51e5068cb9cb93d8763ac3aa7c05b300" exitCode=0 Jul 17 09:33:31.883322 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:31.883299 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" event={"ID":"8257a986-e99b-4089-8b45-2b41e0340103","Type":"ContainerDied","Data":"139169ce7e1032522e3cbe5f0d24257b51e5068cb9cb93d8763ac3aa7c05b300"} Jul 17 09:33:32.891055 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:32.891027 2584 generic.go:356] "Generic (PLEG): container finished" podID="32491371-c26f-4d72-ab8a-308769d06bcf" containerID="2f578510f26015727ab8aca46b37c05d18a093c49850808d8365e30345e4c2a5" exitCode=0 Jul 17 09:33:32.891453 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:32.891110 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" event={"ID":"32491371-c26f-4d72-ab8a-308769d06bcf","Type":"ContainerDied","Data":"2f578510f26015727ab8aca46b37c05d18a093c49850808d8365e30345e4c2a5"} Jul 17 09:33:32.893238 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:32.893216 2584 generic.go:356] "Generic (PLEG): container finished" podID="8257a986-e99b-4089-8b45-2b41e0340103" containerID="7566b3bf38e99551acac1516c325b159a1da164a42b5eeccd67d7f107862bcdd" exitCode=0 Jul 17 09:33:32.893329 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:32.893265 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" event={"ID":"8257a986-e99b-4089-8b45-2b41e0340103","Type":"ContainerDied","Data":"7566b3bf38e99551acac1516c325b159a1da164a42b5eeccd67d7f107862bcdd"} Jul 17 09:33:32.895109 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:32.895087 2584 generic.go:356] "Generic (PLEG): container finished" podID="937498cf-9879-45b0-92ca-3c106854f506" containerID="cfec5268d98c52ec1ce8e82be470773783fe8104bbd7ecdbfb736cdda89e7691" exitCode=0 Jul 17 09:33:32.895200 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:32.895173 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" event={"ID":"937498cf-9879-45b0-92ca-3c106854f506","Type":"ContainerDied","Data":"cfec5268d98c52ec1ce8e82be470773783fe8104bbd7ecdbfb736cdda89e7691"} Jul 17 09:33:33.028515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.028491 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:33.153809 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.153782 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8r8l\" (UniqueName: \"kubernetes.io/projected/a26f9c86-140d-454d-9c10-9acd72107ac0-kube-api-access-s8r8l\") pod \"a26f9c86-140d-454d-9c10-9acd72107ac0\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " Jul 17 09:33:33.153933 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.153826 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-util\") pod \"a26f9c86-140d-454d-9c10-9acd72107ac0\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " Jul 17 09:33:33.153933 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.153847 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-bundle\") pod \"a26f9c86-140d-454d-9c10-9acd72107ac0\" (UID: \"a26f9c86-140d-454d-9c10-9acd72107ac0\") " Jul 17 09:33:33.154343 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.154313 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-bundle" (OuterVolumeSpecName: "bundle") pod "a26f9c86-140d-454d-9c10-9acd72107ac0" (UID: "a26f9c86-140d-454d-9c10-9acd72107ac0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:33:33.155945 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.155917 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26f9c86-140d-454d-9c10-9acd72107ac0-kube-api-access-s8r8l" (OuterVolumeSpecName: "kube-api-access-s8r8l") pod "a26f9c86-140d-454d-9c10-9acd72107ac0" (UID: "a26f9c86-140d-454d-9c10-9acd72107ac0"). InnerVolumeSpecName "kube-api-access-s8r8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:33:33.158869 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.158847 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-util" (OuterVolumeSpecName: "util") pod "a26f9c86-140d-454d-9c10-9acd72107ac0" (UID: "a26f9c86-140d-454d-9c10-9acd72107ac0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:33:33.254848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.254820 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8r8l\" (UniqueName: \"kubernetes.io/projected/a26f9c86-140d-454d-9c10-9acd72107ac0-kube-api-access-s8r8l\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:33.254848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.254845 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-util\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:33.255052 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.254862 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26f9c86-140d-454d-9c10-9acd72107ac0-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:33.903581 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.903547 2584 generic.go:356] "Generic (PLEG): container finished" podID="937498cf-9879-45b0-92ca-3c106854f506" containerID="b0946a3aa3a8baf01f3a44d578b3a6af2c8f4a93c0b61c5f57087325f8a475c7" exitCode=0 Jul 17 09:33:33.903965 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.903636 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" event={"ID":"937498cf-9879-45b0-92ca-3c106854f506","Type":"ContainerDied","Data":"b0946a3aa3a8baf01f3a44d578b3a6af2c8f4a93c0b61c5f57087325f8a475c7"} Jul 17 09:33:33.905447 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.905422 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" event={"ID":"a26f9c86-140d-454d-9c10-9acd72107ac0","Type":"ContainerDied","Data":"1d9f9836ba21bca460b33a743deaea9e2ceea40db7e64884231d86a03429144e"} Jul 17 09:33:33.905561 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.905446 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26" Jul 17 09:33:33.905561 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:33.905463 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d9f9836ba21bca460b33a743deaea9e2ceea40db7e64884231d86a03429144e" Jul 17 09:33:34.037722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.037692 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:34.072544 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.072513 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:34.161667 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.161577 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-util\") pod \"8257a986-e99b-4089-8b45-2b41e0340103\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " Jul 17 09:33:34.161667 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.161640 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-bundle\") pod \"8257a986-e99b-4089-8b45-2b41e0340103\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " Jul 17 09:33:34.161906 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.161706 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgpkk\" (UniqueName: \"kubernetes.io/projected/8257a986-e99b-4089-8b45-2b41e0340103-kube-api-access-qgpkk\") pod \"8257a986-e99b-4089-8b45-2b41e0340103\" (UID: \"8257a986-e99b-4089-8b45-2b41e0340103\") " Jul 17 09:33:34.162350 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.162321 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-bundle" (OuterVolumeSpecName: "bundle") pod "8257a986-e99b-4089-8b45-2b41e0340103" (UID: "8257a986-e99b-4089-8b45-2b41e0340103"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:33:34.163729 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.163704 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8257a986-e99b-4089-8b45-2b41e0340103-kube-api-access-qgpkk" (OuterVolumeSpecName: "kube-api-access-qgpkk") pod "8257a986-e99b-4089-8b45-2b41e0340103" (UID: "8257a986-e99b-4089-8b45-2b41e0340103"). InnerVolumeSpecName "kube-api-access-qgpkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:33:34.167084 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.167051 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-util" (OuterVolumeSpecName: "util") pod "8257a986-e99b-4089-8b45-2b41e0340103" (UID: "8257a986-e99b-4089-8b45-2b41e0340103"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:33:34.262745 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.262708 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-util\") pod \"32491371-c26f-4d72-ab8a-308769d06bcf\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " Jul 17 09:33:34.262931 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.262782 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-bundle\") pod \"32491371-c26f-4d72-ab8a-308769d06bcf\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " Jul 17 09:33:34.262931 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.262827 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h82p6\" (UniqueName: \"kubernetes.io/projected/32491371-c26f-4d72-ab8a-308769d06bcf-kube-api-access-h82p6\") pod \"32491371-c26f-4d72-ab8a-308769d06bcf\" (UID: \"32491371-c26f-4d72-ab8a-308769d06bcf\") " Jul 17 09:33:34.263096 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.263078 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:34.263151 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.263102 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpkk\" (UniqueName: \"kubernetes.io/projected/8257a986-e99b-4089-8b45-2b41e0340103-kube-api-access-qgpkk\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:34.263151 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.263117 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8257a986-e99b-4089-8b45-2b41e0340103-util\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:34.263268 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.263252 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-bundle" (OuterVolumeSpecName: "bundle") pod "32491371-c26f-4d72-ab8a-308769d06bcf" (UID: "32491371-c26f-4d72-ab8a-308769d06bcf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:33:34.264950 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.264925 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32491371-c26f-4d72-ab8a-308769d06bcf-kube-api-access-h82p6" (OuterVolumeSpecName: "kube-api-access-h82p6") pod "32491371-c26f-4d72-ab8a-308769d06bcf" (UID: "32491371-c26f-4d72-ab8a-308769d06bcf"). InnerVolumeSpecName "kube-api-access-h82p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:33:34.268801 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.268775 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-util" (OuterVolumeSpecName: "util") pod "32491371-c26f-4d72-ab8a-308769d06bcf" (UID: "32491371-c26f-4d72-ab8a-308769d06bcf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:33:34.364353 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.364313 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-util\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:34.364353 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.364346 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/32491371-c26f-4d72-ab8a-308769d06bcf-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:34.364353 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.364359 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h82p6\" (UniqueName: \"kubernetes.io/projected/32491371-c26f-4d72-ab8a-308769d06bcf-kube-api-access-h82p6\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:34.912686 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.912656 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" Jul 17 09:33:34.913171 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.912649 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd" event={"ID":"32491371-c26f-4d72-ab8a-308769d06bcf","Type":"ContainerDied","Data":"3d59427a1db1681e467379b6b774b2b69a51e12e017b789c4e881cb7d9cd5897"} Jul 17 09:33:34.913171 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.912803 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d59427a1db1681e467379b6b774b2b69a51e12e017b789c4e881cb7d9cd5897" Jul 17 09:33:34.914934 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.914914 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" Jul 17 09:33:34.914934 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.914910 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j" event={"ID":"8257a986-e99b-4089-8b45-2b41e0340103","Type":"ContainerDied","Data":"08a6641593d522655d8003c57fbe96f2d2adc50335eb05168d0183ebcbdd8646"} Jul 17 09:33:34.915091 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:34.914961 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08a6641593d522655d8003c57fbe96f2d2adc50335eb05168d0183ebcbdd8646" Jul 17 09:33:35.043082 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.043059 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:35.171866 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.171786 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-util\") pod \"937498cf-9879-45b0-92ca-3c106854f506\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " Jul 17 09:33:35.171866 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.171841 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-bundle\") pod \"937498cf-9879-45b0-92ca-3c106854f506\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " Jul 17 09:33:35.172088 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.171970 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ft9b\" (UniqueName: \"kubernetes.io/projected/937498cf-9879-45b0-92ca-3c106854f506-kube-api-access-5ft9b\") pod \"937498cf-9879-45b0-92ca-3c106854f506\" (UID: \"937498cf-9879-45b0-92ca-3c106854f506\") " Jul 17 09:33:35.172437 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.172387 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-bundle" (OuterVolumeSpecName: "bundle") pod "937498cf-9879-45b0-92ca-3c106854f506" (UID: "937498cf-9879-45b0-92ca-3c106854f506"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:33:35.173958 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.173938 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937498cf-9879-45b0-92ca-3c106854f506-kube-api-access-5ft9b" (OuterVolumeSpecName: "kube-api-access-5ft9b") pod "937498cf-9879-45b0-92ca-3c106854f506" (UID: "937498cf-9879-45b0-92ca-3c106854f506"). InnerVolumeSpecName "kube-api-access-5ft9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:33:35.176877 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.176853 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-util" (OuterVolumeSpecName: "util") pod "937498cf-9879-45b0-92ca-3c106854f506" (UID: "937498cf-9879-45b0-92ca-3c106854f506"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:33:35.273034 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.272995 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ft9b\" (UniqueName: \"kubernetes.io/projected/937498cf-9879-45b0-92ca-3c106854f506-kube-api-access-5ft9b\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:35.273034 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.273033 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-util\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:35.273034 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.273042 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/937498cf-9879-45b0-92ca-3c106854f506-bundle\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:33:35.921836 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.921807 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" Jul 17 09:33:35.921836 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.921823 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259" event={"ID":"937498cf-9879-45b0-92ca-3c106854f506","Type":"ContainerDied","Data":"f6cd3ef3ce1c86df7346976fe9b6cb9baefc7c9f9c69dec73105e11c0fb6657d"} Jul 17 09:33:35.922250 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:33:35.921863 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6cd3ef3ce1c86df7346976fe9b6cb9baefc7c9f9c69dec73105e11c0fb6657d" Jul 17 09:35:22.687991 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.687953 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c5c6b98f-656qz"] Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688445 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerName="util" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688466 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerName="util" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688483 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8257a986-e99b-4089-8b45-2b41e0340103" containerName="extract" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688491 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8257a986-e99b-4089-8b45-2b41e0340103" containerName="extract" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688504 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32491371-c26f-4d72-ab8a-308769d06bcf" containerName="util" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688513 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32491371-c26f-4d72-ab8a-308769d06bcf" containerName="util" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688521 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32491371-c26f-4d72-ab8a-308769d06bcf" containerName="extract" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688529 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32491371-c26f-4d72-ab8a-308769d06bcf" containerName="extract" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688545 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerName="extract" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688553 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerName="extract" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688566 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937498cf-9879-45b0-92ca-3c106854f506" containerName="extract" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688574 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="937498cf-9879-45b0-92ca-3c106854f506" containerName="extract" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688589 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8257a986-e99b-4089-8b45-2b41e0340103" containerName="pull" Jul 17 09:35:22.688590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688597 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8257a986-e99b-4089-8b45-2b41e0340103" containerName="pull" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688610 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32491371-c26f-4d72-ab8a-308769d06bcf" containerName="pull" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688617 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="32491371-c26f-4d72-ab8a-308769d06bcf" containerName="pull" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688633 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937498cf-9879-45b0-92ca-3c106854f506" containerName="util" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688641 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="937498cf-9879-45b0-92ca-3c106854f506" containerName="util" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688653 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerName="pull" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688660 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerName="pull" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688671 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8257a986-e99b-4089-8b45-2b41e0340103" containerName="util" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688679 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8257a986-e99b-4089-8b45-2b41e0340103" containerName="util" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688689 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937498cf-9879-45b0-92ca-3c106854f506" containerName="pull" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688697 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="937498cf-9879-45b0-92ca-3c106854f506" containerName="pull" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688783 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="937498cf-9879-45b0-92ca-3c106854f506" containerName="extract" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688794 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8257a986-e99b-4089-8b45-2b41e0340103" containerName="extract" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688805 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26f9c86-140d-454d-9c10-9acd72107ac0" containerName="extract" Jul 17 09:35:22.689316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.688813 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="32491371-c26f-4d72-ab8a-308769d06bcf" containerName="extract" Jul 17 09:35:22.692022 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.692003 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:22.694765 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.694743 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 09:35:22.694867 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.694751 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jc8mw\"" Jul 17 09:35:22.695977 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.695960 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 09:35:22.700685 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.700660 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5c6b98f-656qz"] Jul 17 09:35:22.731036 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.731012 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrxk2\" (UniqueName: \"kubernetes.io/projected/6aa09ad5-5515-48a9-889d-5caa72b508f7-kube-api-access-nrxk2\") pod \"maas-controller-6c5c6b98f-656qz\" (UID: \"6aa09ad5-5515-48a9-889d-5caa72b508f7\") " pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:22.731142 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.731056 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6aa09ad5-5515-48a9-889d-5caa72b508f7-webhook-cert\") pod \"maas-controller-6c5c6b98f-656qz\" (UID: \"6aa09ad5-5515-48a9-889d-5caa72b508f7\") " pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:22.832024 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.832000 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrxk2\" (UniqueName: \"kubernetes.io/projected/6aa09ad5-5515-48a9-889d-5caa72b508f7-kube-api-access-nrxk2\") pod \"maas-controller-6c5c6b98f-656qz\" (UID: \"6aa09ad5-5515-48a9-889d-5caa72b508f7\") " pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:22.832152 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.832065 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6aa09ad5-5515-48a9-889d-5caa72b508f7-webhook-cert\") pod \"maas-controller-6c5c6b98f-656qz\" (UID: \"6aa09ad5-5515-48a9-889d-5caa72b508f7\") " pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:22.834444 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.834423 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6aa09ad5-5515-48a9-889d-5caa72b508f7-webhook-cert\") pod \"maas-controller-6c5c6b98f-656qz\" (UID: \"6aa09ad5-5515-48a9-889d-5caa72b508f7\") " pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:22.839863 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:22.839846 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrxk2\" (UniqueName: \"kubernetes.io/projected/6aa09ad5-5515-48a9-889d-5caa72b508f7-kube-api-access-nrxk2\") pod \"maas-controller-6c5c6b98f-656qz\" (UID: \"6aa09ad5-5515-48a9-889d-5caa72b508f7\") " pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:23.003338 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:23.003271 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:23.129624 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:23.129601 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5c6b98f-656qz"] Jul 17 09:35:23.131437 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:35:23.131410 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa09ad5_5515_48a9_889d_5caa72b508f7.slice/crio-e81ef987d256414e12d3938b1277f4d0acc08e53850bff4f4e4c7c1f35e763ff WatchSource:0}: Error finding container e81ef987d256414e12d3938b1277f4d0acc08e53850bff4f4e4c7c1f35e763ff: Status 404 returned error can't find the container with id e81ef987d256414e12d3938b1277f4d0acc08e53850bff4f4e4c7c1f35e763ff Jul 17 09:35:23.436823 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:23.436787 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5c6b98f-656qz" event={"ID":"6aa09ad5-5515-48a9-889d-5caa72b508f7","Type":"ContainerStarted","Data":"e81ef987d256414e12d3938b1277f4d0acc08e53850bff4f4e4c7c1f35e763ff"} Jul 17 09:35:26.453621 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:26.453582 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5c6b98f-656qz" event={"ID":"6aa09ad5-5515-48a9-889d-5caa72b508f7","Type":"ContainerStarted","Data":"564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55"} Jul 17 09:35:26.453991 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:26.453650 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:26.474122 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:26.474077 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c5c6b98f-656qz" podStartSLOduration=2.21093542 podStartE2EDuration="4.474063151s" totalImagesPullingTime="2.263127731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:35:26.472175418 +0000 UTC m=+509.011374377" watchObservedRunningTime="2026-07-17 09:35:26.474063151 +0000 UTC m=+509.013262147" Jul 17 09:35:37.460523 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:37.460495 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:35:57.070066 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.069987 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6957c666b-jklr9"] Jul 17 09:35:57.072859 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.072838 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.075977 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.075955 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 17 09:35:57.076177 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.075957 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 09:35:57.076262 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.076000 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bl7qx\"" Jul 17 09:35:57.077091 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.077071 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 09:35:57.085066 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.085041 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6957c666b-jklr9"] Jul 17 09:35:57.120792 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.120765 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcd32724-ea72-48a2-ab3e-239e7345db38-maas-api-tls\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.120955 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.120802 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcd32724-ea72-48a2-ab3e-239e7345db38-openshift-service-ca\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.120955 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.120826 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxht7\" (UniqueName: \"kubernetes.io/projected/fcd32724-ea72-48a2-ab3e-239e7345db38-kube-api-access-bxht7\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.221510 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.221473 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxht7\" (UniqueName: \"kubernetes.io/projected/fcd32724-ea72-48a2-ab3e-239e7345db38-kube-api-access-bxht7\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.221697 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.221564 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcd32724-ea72-48a2-ab3e-239e7345db38-maas-api-tls\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.221697 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.221593 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcd32724-ea72-48a2-ab3e-239e7345db38-openshift-service-ca\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.221814 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:35:57.221737 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 17 09:35:57.221866 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:35:57.221854 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd32724-ea72-48a2-ab3e-239e7345db38-maas-api-tls podName:fcd32724-ea72-48a2-ab3e-239e7345db38 nodeName:}" failed. No retries permitted until 2026-07-17 09:35:57.721830633 +0000 UTC m=+540.261029579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fcd32724-ea72-48a2-ab3e-239e7345db38-maas-api-tls") pod "maas-api-6957c666b-jklr9" (UID: "fcd32724-ea72-48a2-ab3e-239e7345db38") : secret "maas-api-serving-cert" not found Jul 17 09:35:57.222221 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.222201 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcd32724-ea72-48a2-ab3e-239e7345db38-openshift-service-ca\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.230267 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.230242 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxht7\" (UniqueName: \"kubernetes.io/projected/fcd32724-ea72-48a2-ab3e-239e7345db38-kube-api-access-bxht7\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.727521 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.727490 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcd32724-ea72-48a2-ab3e-239e7345db38-maas-api-tls\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.729977 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.729947 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcd32724-ea72-48a2-ab3e-239e7345db38-maas-api-tls\") pod \"maas-api-6957c666b-jklr9\" (UID: \"fcd32724-ea72-48a2-ab3e-239e7345db38\") " pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:57.988538 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.988468 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bl7qx\"" Jul 17 09:35:57.996226 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:57.996200 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:35:58.126308 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:58.126284 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6957c666b-jklr9"] Jul 17 09:35:58.127917 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:35:58.127890 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd32724_ea72_48a2_ab3e_239e7345db38.slice/crio-56d94cee96ead8811cfdfa58e1b7cc8b3b56b5e694cc5278395c355c5ec2eb32 WatchSource:0}: Error finding container 56d94cee96ead8811cfdfa58e1b7cc8b3b56b5e694cc5278395c355c5ec2eb32: Status 404 returned error can't find the container with id 56d94cee96ead8811cfdfa58e1b7cc8b3b56b5e694cc5278395c355c5ec2eb32 Jul 17 09:35:58.611481 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:35:58.611452 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" event={"ID":"fcd32724-ea72-48a2-ab3e-239e7345db38","Type":"ContainerStarted","Data":"56d94cee96ead8811cfdfa58e1b7cc8b3b56b5e694cc5278395c355c5ec2eb32"} Jul 17 09:36:00.623811 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:00.623777 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" event={"ID":"fcd32724-ea72-48a2-ab3e-239e7345db38","Type":"ContainerStarted","Data":"a0229343142a99ccdd79cbe7ff658b7cf0da3c687c3f7076185ea322fad7599d"} Jul 17 09:36:00.624183 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:00.623848 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:36:00.642746 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:00.642702 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" podStartSLOduration=2.200979536 podStartE2EDuration="3.642686957s" totalImagesPullingTime="1.441707421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:36:00.641040626 +0000 UTC m=+543.180239593" watchObservedRunningTime="2026-07-17 09:36:00.642686957 +0000 UTC m=+543.181885961" Jul 17 09:36:06.635959 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:06.635932 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6957c666b-jklr9" Jul 17 09:36:33.455926 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.455886 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj"] Jul 17 09:36:33.458827 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.458808 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.462575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.462549 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-thd5v\"" Jul 17 09:36:33.462575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.462558 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 17 09:36:33.462575 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.462548 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 09:36:33.462802 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.462629 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 09:36:33.469298 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.469278 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj"] Jul 17 09:36:33.635133 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.635101 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.635296 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.635148 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2zj6\" (UniqueName: \"kubernetes.io/projected/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-kube-api-access-d2zj6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.635296 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.635170 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.635296 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.635233 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.635296 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.635281 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.635468 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.635351 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736098 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.735998 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736098 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.736058 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zj6\" (UniqueName: \"kubernetes.io/projected/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-kube-api-access-d2zj6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736098 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.736081 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736098 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.736108 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736456 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.736136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736456 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.736185 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736456 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.736429 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736560 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.736526 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.736560 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.736534 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.738365 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.738346 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.738559 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.738543 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.744221 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.744199 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zj6\" (UniqueName: \"kubernetes.io/projected/aa31d65e-8aa0-4dc0-a348-e57adf5e2f18-kube-api-access-d2zj6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2cvvj\" (UID: \"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.771228 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.771208 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:33.907079 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:33.907047 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj"] Jul 17 09:36:33.908696 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:36:33.908671 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa31d65e_8aa0_4dc0_a348_e57adf5e2f18.slice/crio-3b72f74d9d90ee0c600803e3247e83a14ffae6488b733bf1e338760ca51a22ec WatchSource:0}: Error finding container 3b72f74d9d90ee0c600803e3247e83a14ffae6488b733bf1e338760ca51a22ec: Status 404 returned error can't find the container with id 3b72f74d9d90ee0c600803e3247e83a14ffae6488b733bf1e338760ca51a22ec Jul 17 09:36:34.800320 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:34.800280 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" event={"ID":"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18","Type":"ContainerStarted","Data":"3b72f74d9d90ee0c600803e3247e83a14ffae6488b733bf1e338760ca51a22ec"} Jul 17 09:36:35.806677 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:35.806589 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" event={"ID":"aa31d65e-8aa0-4dc0-a348-e57adf5e2f18","Type":"ContainerStarted","Data":"153ef4b38c691f30bed9c42d04e2febbe057ce555d324deb2bef3b5183b26eb0"} Jul 17 09:36:35.826296 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:35.826251 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" podStartSLOduration=1.354497674 podStartE2EDuration="2.826236148s" totalImagesPullingTime="1.471738474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:36:35.824761516 +0000 UTC m=+578.363960497" watchObservedRunningTime="2026-07-17 09:36:35.826236148 +0000 UTC m=+578.365435113" Jul 17 09:36:43.772230 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:43.772194 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:43.772661 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:43.772264 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:43.784751 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:43.784728 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:43.856826 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:43.856801 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2cvvj" Jul 17 09:36:57.908950 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:57.908922 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:36:57.914046 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:36:57.914023 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:37:15.275378 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.275338 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls"] Jul 17 09:37:15.281432 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.281412 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.284025 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.284005 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 17 09:37:15.289307 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.289268 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls"] Jul 17 09:37:15.381140 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.381108 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpdl7\" (UniqueName: \"kubernetes.io/projected/0387d70e-160c-456c-b149-0453d439bb01-kube-api-access-tpdl7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.381314 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.381155 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.381314 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.381214 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.381314 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.381243 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0387d70e-160c-456c-b149-0453d439bb01-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.381510 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.381319 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.381510 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.381375 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482302 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482266 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpdl7\" (UniqueName: \"kubernetes.io/projected/0387d70e-160c-456c-b149-0453d439bb01-kube-api-access-tpdl7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482492 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482315 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482492 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482341 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482492 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482365 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0387d70e-160c-456c-b149-0453d439bb01-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482492 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482391 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482492 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482431 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482811 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482793 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482865 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482848 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.482904 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.482863 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.484659 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.484635 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0387d70e-160c-456c-b149-0453d439bb01-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.484903 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.484886 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0387d70e-160c-456c-b149-0453d439bb01-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.490547 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.490521 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpdl7\" (UniqueName: \"kubernetes.io/projected/0387d70e-160c-456c-b149-0453d439bb01-kube-api-access-tpdl7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls\" (UID: \"0387d70e-160c-456c-b149-0453d439bb01\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.592538 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.592503 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:15.724345 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:15.724310 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls"] Jul 17 09:37:15.725317 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:37:15.725294 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0387d70e_160c_456c_b149_0453d439bb01.slice/crio-b004e6fab31466522759256fb6d07e3a298501797a86472ed6a5f968f1aae45c WatchSource:0}: Error finding container b004e6fab31466522759256fb6d07e3a298501797a86472ed6a5f968f1aae45c: Status 404 returned error can't find the container with id b004e6fab31466522759256fb6d07e3a298501797a86472ed6a5f968f1aae45c Jul 17 09:37:16.007916 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:16.007879 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" event={"ID":"0387d70e-160c-456c-b149-0453d439bb01","Type":"ContainerStarted","Data":"8c60ab6cb94ac09071c189c5b7d0d82b087a05b6298c427317f34408e2827d07"} Jul 17 09:37:16.008057 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:16.007943 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" event={"ID":"0387d70e-160c-456c-b149-0453d439bb01","Type":"ContainerStarted","Data":"b004e6fab31466522759256fb6d07e3a298501797a86472ed6a5f968f1aae45c"} Jul 17 09:37:16.025576 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:16.025538 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" podStartSLOduration=0.822760368 podStartE2EDuration="1.025522972s" totalImagesPullingTime="202.762604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:37:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:37:16.023695059 +0000 UTC m=+618.562894019" watchObservedRunningTime="2026-07-17 09:37:16.025522972 +0000 UTC m=+618.564721937" Jul 17 09:37:22.257221 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.257131 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t"] Jul 17 09:37:22.260489 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.260467 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.262929 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.262912 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 17 09:37:22.304740 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.304715 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t"] Jul 17 09:37:22.342310 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.342282 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.342485 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.342315 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.342485 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.342344 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.342485 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.342464 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.342595 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.342506 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42brz\" (UniqueName: \"kubernetes.io/projected/a674834a-dedc-4f56-a554-557dabad2233-kube-api-access-42brz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.342595 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.342529 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a674834a-dedc-4f56-a554-557dabad2233-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.443515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.443477 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.443515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.443513 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.443752 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.443542 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.443752 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.443573 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.443752 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.443601 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42brz\" (UniqueName: \"kubernetes.io/projected/a674834a-dedc-4f56-a554-557dabad2233-kube-api-access-42brz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.443752 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.443623 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a674834a-dedc-4f56-a554-557dabad2233-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.444091 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.444069 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.444212 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.444100 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.444276 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.444253 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.446272 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.446247 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a674834a-dedc-4f56-a554-557dabad2233-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.446488 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.446461 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a674834a-dedc-4f56-a554-557dabad2233-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.451585 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.451560 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42brz\" (UniqueName: \"kubernetes.io/projected/a674834a-dedc-4f56-a554-557dabad2233-kube-api-access-42brz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gtm4t\" (UID: \"a674834a-dedc-4f56-a554-557dabad2233\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.570950 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.570881 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:22.702045 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.702019 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t"] Jul 17 09:37:22.703050 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:37:22.703026 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda674834a_dedc_4f56_a554_557dabad2233.slice/crio-4646f5f65f2696e43cfc0b33206834e33fe78a7066dee90e9590880868e3f7ed WatchSource:0}: Error finding container 4646f5f65f2696e43cfc0b33206834e33fe78a7066dee90e9590880868e3f7ed: Status 404 returned error can't find the container with id 4646f5f65f2696e43cfc0b33206834e33fe78a7066dee90e9590880868e3f7ed Jul 17 09:37:22.704322 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:22.704300 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 09:37:23.043544 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:23.043501 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" event={"ID":"a674834a-dedc-4f56-a554-557dabad2233","Type":"ContainerStarted","Data":"554d89146c6a0c8d469f5a2fc4f3db1daea248c742d9d4adf37b7d2f1b198e1c"} Jul 17 09:37:23.043705 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:23.043554 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" event={"ID":"a674834a-dedc-4f56-a554-557dabad2233","Type":"ContainerStarted","Data":"4646f5f65f2696e43cfc0b33206834e33fe78a7066dee90e9590880868e3f7ed"} Jul 17 09:37:23.060377 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:23.060328 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" podStartSLOduration=0.830597188 podStartE2EDuration="1.060312654s" totalImagesPullingTime="229.715466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:37:23.059672234 +0000 UTC m=+625.598871225" watchObservedRunningTime="2026-07-17 09:37:23.060312654 +0000 UTC m=+625.599511620" Jul 17 09:37:25.593559 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:25.593516 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:25.593988 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:25.593625 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:25.606123 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:25.606096 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:26.068589 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:26.068513 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls" Jul 17 09:37:32.571448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:32.571388 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:32.571865 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:32.571478 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:32.583919 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:32.583896 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:37:33.104376 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:37:33.104341 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gtm4t" Jul 17 09:39:11.131968 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.131872 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5c6b98f-656qz"] Jul 17 09:39:11.132499 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.132097 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c5c6b98f-656qz" podUID="6aa09ad5-5515-48a9-889d-5caa72b508f7" containerName="manager" containerID="cri-o://564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55" gracePeriod=30 Jul 17 09:39:11.388840 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.388776 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:39:11.525743 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.525710 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6aa09ad5-5515-48a9-889d-5caa72b508f7-webhook-cert\") pod \"6aa09ad5-5515-48a9-889d-5caa72b508f7\" (UID: \"6aa09ad5-5515-48a9-889d-5caa72b508f7\") " Jul 17 09:39:11.525896 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.525767 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrxk2\" (UniqueName: \"kubernetes.io/projected/6aa09ad5-5515-48a9-889d-5caa72b508f7-kube-api-access-nrxk2\") pod \"6aa09ad5-5515-48a9-889d-5caa72b508f7\" (UID: \"6aa09ad5-5515-48a9-889d-5caa72b508f7\") " Jul 17 09:39:11.527861 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.527837 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa09ad5-5515-48a9-889d-5caa72b508f7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6aa09ad5-5515-48a9-889d-5caa72b508f7" (UID: "6aa09ad5-5515-48a9-889d-5caa72b508f7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 09:39:11.528021 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.528001 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa09ad5-5515-48a9-889d-5caa72b508f7-kube-api-access-nrxk2" (OuterVolumeSpecName: "kube-api-access-nrxk2") pod "6aa09ad5-5515-48a9-889d-5caa72b508f7" (UID: "6aa09ad5-5515-48a9-889d-5caa72b508f7"). InnerVolumeSpecName "kube-api-access-nrxk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:39:11.608834 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.608796 2584 generic.go:356] "Generic (PLEG): container finished" podID="6aa09ad5-5515-48a9-889d-5caa72b508f7" containerID="564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55" exitCode=0 Jul 17 09:39:11.608998 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.608848 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5c6b98f-656qz" Jul 17 09:39:11.608998 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.608885 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5c6b98f-656qz" event={"ID":"6aa09ad5-5515-48a9-889d-5caa72b508f7","Type":"ContainerDied","Data":"564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55"} Jul 17 09:39:11.608998 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.608955 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5c6b98f-656qz" event={"ID":"6aa09ad5-5515-48a9-889d-5caa72b508f7","Type":"ContainerDied","Data":"e81ef987d256414e12d3938b1277f4d0acc08e53850bff4f4e4c7c1f35e763ff"} Jul 17 09:39:11.608998 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.608977 2584 scope.go:117] "RemoveContainer" containerID="564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55" Jul 17 09:39:11.618178 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.618162 2584 scope.go:117] "RemoveContainer" containerID="564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55" Jul 17 09:39:11.618498 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:39:11.618474 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55\": container with ID starting with 564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55 not found: ID does not exist" containerID="564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55" Jul 17 09:39:11.618569 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.618504 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55"} err="failed to get container status \"564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55\": rpc error: code = NotFound desc = could not find container \"564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55\": container with ID starting with 564a09bf146fb7f5c381654cfecc2a8cf54f474bf401e7df71816ce599f4ca55 not found: ID does not exist" Jul 17 09:39:11.626767 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.626746 2584 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6aa09ad5-5515-48a9-889d-5caa72b508f7-webhook-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:39:11.626848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.626768 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrxk2\" (UniqueName: \"kubernetes.io/projected/6aa09ad5-5515-48a9-889d-5caa72b508f7-kube-api-access-nrxk2\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:39:11.629969 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.629945 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5c6b98f-656qz"] Jul 17 09:39:11.634266 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.634244 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c5c6b98f-656qz"] Jul 17 09:39:11.998750 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:39:11.998715 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa09ad5-5515-48a9-889d-5caa72b508f7" path="/var/lib/kubelet/pods/6aa09ad5-5515-48a9-889d-5caa72b508f7/volumes" Jul 17 09:41:57.945916 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:41:57.945845 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:41:57.948448 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:41:57.948423 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:46:57.989999 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:46:57.989962 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:46:57.990535 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:46:57.990085 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:51:58.029170 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:51:58.029139 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:51:58.029861 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:51:58.029844 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:52:21.085929 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.085853 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228"] Jul 17 09:52:21.086416 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.086242 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa09ad5-5515-48a9-889d-5caa72b508f7" containerName="manager" Jul 17 09:52:21.086416 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.086254 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa09ad5-5515-48a9-889d-5caa72b508f7" containerName="manager" Jul 17 09:52:21.086416 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.086357 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa09ad5-5515-48a9-889d-5caa72b508f7" containerName="manager" Jul 17 09:52:21.089507 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.089489 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.092703 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.092680 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3fc19893-openshift-default-dockercfg-llcjg\"" Jul 17 09:52:21.117510 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.117479 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228"] Jul 17 09:52:21.228173 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228138 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-envoy\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.228173 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228178 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblpt\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-kube-api-access-fblpt\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.228488 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228266 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-token\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.228488 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228319 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-certs\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.228488 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228367 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-podinfo\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.228488 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228429 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-credential-socket\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.228488 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228456 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istiod-ca-cert\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.228661 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228492 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-socket\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.228661 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.228545 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-data\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.329854 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.329801 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-podinfo\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.329854 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.329855 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-credential-socket\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.329854 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.329871 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istiod-ca-cert\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330136 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.329942 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-socket\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330136 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.329988 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-data\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330136 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330078 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-envoy\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330280 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330259 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblpt\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-kube-api-access-fblpt\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330349 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330323 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-socket\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330553 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330348 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-token\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330553 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330384 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-credential-socket\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330553 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330510 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-certs\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330553 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330386 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-data\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330740 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330605 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istiod-ca-cert\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.330805 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.330787 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-certs\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.332783 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.332768 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-envoy\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.333103 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.333081 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-podinfo\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.338945 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.338926 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-token\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.339497 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.339477 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblpt\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-kube-api-access-fblpt\") pod \"e2e-ait-3fc19893-openshift-default-68f7994775-4c228\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.401039 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.401008 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:21.534649 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.534619 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228"] Jul 17 09:52:21.535628 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:52:21.535604 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e6aa51_f865_4401_ab5f_6ed4c1653a37.slice/crio-4825a9e062df90eccd2250403a0602a5aede2f944b499a9e4e67610d470320af WatchSource:0}: Error finding container 4825a9e062df90eccd2250403a0602a5aede2f944b499a9e4e67610d470320af: Status 404 returned error can't find the container with id 4825a9e062df90eccd2250403a0602a5aede2f944b499a9e4e67610d470320af Jul 17 09:52:21.537667 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.537636 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:52:21.537768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.537697 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:52:21.537768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.537724 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:52:21.677362 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.677267 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" event={"ID":"b2e6aa51-f865-4401-ab5f-6ed4c1653a37","Type":"ContainerStarted","Data":"6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d"} Jul 17 09:52:21.677362 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.677320 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" event={"ID":"b2e6aa51-f865-4401-ab5f-6ed4c1653a37","Type":"ContainerStarted","Data":"4825a9e062df90eccd2250403a0602a5aede2f944b499a9e4e67610d470320af"} Jul 17 09:52:21.699969 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:21.699905 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" podStartSLOduration=0.699882638 podStartE2EDuration="699.882638ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:52:21.695628102 +0000 UTC m=+1524.234827060" watchObservedRunningTime="2026-07-17 09:52:21.699882638 +0000 UTC m=+1524.239081605" Jul 17 09:52:22.396048 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.396011 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n"] Jul 17 09:52:22.400030 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.400005 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.401262 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.401235 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:22.402904 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.402881 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3fc19893-serving-cert\"" Jul 17 09:52:22.406584 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.406564 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:22.412986 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.412955 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n"] Jul 17 09:52:22.541663 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.541609 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b692649-cf69-4039-8130-6f0e0e778702-openshift-service-ca\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.541856 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.541693 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvbj7\" (UniqueName: \"kubernetes.io/projected/3b692649-cf69-4039-8130-6f0e0e778702-kube-api-access-mvbj7\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.541856 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.541762 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.642861 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.642806 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.643050 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:22.642978 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-serving-cert: secret "maas-api-e2e-ait-3fc19893-serving-cert" not found Jul 17 09:52:22.643050 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.642979 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b692649-cf69-4039-8130-6f0e0e778702-openshift-service-ca\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.643211 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:22.643055 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls podName:3b692649-cf69-4039-8130-6f0e0e778702 nodeName:}" failed. No retries permitted until 2026-07-17 09:52:23.143033339 +0000 UTC m=+1525.682232285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls") pod "maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" (UID: "3b692649-cf69-4039-8130-6f0e0e778702") : secret "maas-api-e2e-ait-3fc19893-serving-cert" not found Jul 17 09:52:22.643211 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.643134 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvbj7\" (UniqueName: \"kubernetes.io/projected/3b692649-cf69-4039-8130-6f0e0e778702-kube-api-access-mvbj7\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.643811 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.643785 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b692649-cf69-4039-8130-6f0e0e778702-openshift-service-ca\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.654432 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.654331 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvbj7\" (UniqueName: \"kubernetes.io/projected/3b692649-cf69-4039-8130-6f0e0e778702-kube-api-access-mvbj7\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:22.683081 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.683047 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:22.684118 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:22.684094 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:23.148759 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:23.148714 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:23.151275 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:23.151244 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls\") pod \"maas-api-e2e-ait-3fc19893-7d796947b9-92v7n\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:23.313853 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:23.313821 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:23.450572 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:23.450542 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n"] Jul 17 09:52:23.451818 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:52:23.451792 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b692649_cf69_4039_8130_6f0e0e778702.slice/crio-7dd03e5aad8577f4737b4832aca6928a09be9a62addb88f22f58ec620ffddbaf WatchSource:0}: Error finding container 7dd03e5aad8577f4737b4832aca6928a09be9a62addb88f22f58ec620ffddbaf: Status 404 returned error can't find the container with id 7dd03e5aad8577f4737b4832aca6928a09be9a62addb88f22f58ec620ffddbaf Jul 17 09:52:23.459002 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:23.453490 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 09:52:23.689960 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:23.689921 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerStarted","Data":"7dd03e5aad8577f4737b4832aca6928a09be9a62addb88f22f58ec620ffddbaf"} Jul 17 09:52:24.697036 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:24.697000 2584 generic.go:356] "Generic (PLEG): container finished" podID="3b692649-cf69-4039-8130-6f0e0e778702" containerID="002556201a851b2120c64ea06ba0d1c5e73761667537d54260b1b998db618ce8" exitCode=1 Jul 17 09:52:24.697515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:24.697096 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerDied","Data":"002556201a851b2120c64ea06ba0d1c5e73761667537d54260b1b998db618ce8"} Jul 17 09:52:24.697515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:24.697286 2584 scope.go:117] "RemoveContainer" containerID="002556201a851b2120c64ea06ba0d1c5e73761667537d54260b1b998db618ce8" Jul 17 09:52:25.704350 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:25.704257 2584 generic.go:356] "Generic (PLEG): container finished" podID="3b692649-cf69-4039-8130-6f0e0e778702" containerID="508e34859393c3cccb2340633d9f9bd0720cc1ae47607244fd5645ac131faee3" exitCode=1 Jul 17 09:52:25.704723 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:25.704350 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerDied","Data":"508e34859393c3cccb2340633d9f9bd0720cc1ae47607244fd5645ac131faee3"} Jul 17 09:52:25.704723 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:25.704446 2584 scope.go:117] "RemoveContainer" containerID="002556201a851b2120c64ea06ba0d1c5e73761667537d54260b1b998db618ce8" Jul 17 09:52:25.704723 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:25.704655 2584 scope.go:117] "RemoveContainer" containerID="508e34859393c3cccb2340633d9f9bd0720cc1ae47607244fd5645ac131faee3" Jul 17 09:52:25.704954 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:25.704932 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:52:26.711773 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:26.711741 2584 scope.go:117] "RemoveContainer" containerID="508e34859393c3cccb2340633d9f9bd0720cc1ae47607244fd5645ac131faee3" Jul 17 09:52:26.712242 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:26.712013 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:52:28.314247 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:28.314208 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:28.314702 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:28.314665 2584 scope.go:117] "RemoveContainer" containerID="508e34859393c3cccb2340633d9f9bd0720cc1ae47607244fd5645ac131faee3" Jul 17 09:52:28.314860 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:28.314837 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:52:33.313993 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:33.313948 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:33.314554 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:33.314507 2584 scope.go:117] "RemoveContainer" containerID="508e34859393c3cccb2340633d9f9bd0720cc1ae47607244fd5645ac131faee3" Jul 17 09:52:33.314761 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:33.314733 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:52:33.750604 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:33.750572 2584 generic.go:356] "Generic (PLEG): container finished" podID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerID="6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d" exitCode=0 Jul 17 09:52:33.750793 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:33.750654 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" event={"ID":"b2e6aa51-f865-4401-ab5f-6ed4c1653a37","Type":"ContainerDied","Data":"6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d"} Jul 17 09:52:33.751120 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:33.751093 2584 scope.go:117] "RemoveContainer" containerID="6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d" Jul 17 09:52:33.752119 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:33.752093 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:52:33.752171 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:33.752159 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:52:33.752223 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:33.752211 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:52:34.761952 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:34.761909 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" event={"ID":"b2e6aa51-f865-4401-ab5f-6ed4c1653a37","Type":"ContainerStarted","Data":"35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d"} Jul 17 09:52:35.401927 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:35.401889 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:35.403157 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:35.403125 2584 patch_prober.go:28] interesting pod/e2e-ait-3fc19893-openshift-default-68f7994775-4c228 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 17 09:52:35.403281 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:35.403180 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 17 09:52:36.401470 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:36.401426 2584 patch_prober.go:28] interesting pod/e2e-ait-3fc19893-openshift-default-68f7994775-4c228 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 17 09:52:36.401895 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:36.401495 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 17 09:52:37.402470 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:37.402435 2584 patch_prober.go:28] interesting pod/e2e-ait-3fc19893-openshift-default-68f7994775-4c228 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 17 09:52:37.402959 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:37.402930 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 17 09:52:37.683015 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:37.682920 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:38.401589 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:38.401546 2584 patch_prober.go:28] interesting pod/e2e-ait-3fc19893-openshift-default-68f7994775-4c228 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 17 09:52:38.401784 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:38.401608 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 17 09:52:39.405774 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:39.405744 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:39.406935 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:39.406915 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:52:44.993157 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:44.993109 2584 scope.go:117] "RemoveContainer" containerID="508e34859393c3cccb2340633d9f9bd0720cc1ae47607244fd5645ac131faee3" Jul 17 09:52:45.823683 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:45.823647 2584 generic.go:356] "Generic (PLEG): container finished" podID="3b692649-cf69-4039-8130-6f0e0e778702" containerID="a9e828b67c6d1482d22f98a1795eb4a443ceb054f1d36d7b587e1679a8fed3b5" exitCode=1 Jul 17 09:52:45.823876 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:45.823680 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerDied","Data":"a9e828b67c6d1482d22f98a1795eb4a443ceb054f1d36d7b587e1679a8fed3b5"} Jul 17 09:52:45.823876 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:45.823755 2584 scope.go:117] "RemoveContainer" containerID="508e34859393c3cccb2340633d9f9bd0720cc1ae47607244fd5645ac131faee3" Jul 17 09:52:45.824139 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:45.824119 2584 scope.go:117] "RemoveContainer" containerID="a9e828b67c6d1482d22f98a1795eb4a443ceb054f1d36d7b587e1679a8fed3b5" Jul 17 09:52:45.824353 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:45.824328 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:52:48.313969 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:48.313928 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:48.314374 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:48.314320 2584 scope.go:117] "RemoveContainer" containerID="a9e828b67c6d1482d22f98a1795eb4a443ceb054f1d36d7b587e1679a8fed3b5" Jul 17 09:52:48.314546 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:48.314526 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:52:53.314144 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:53.314102 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:52:53.314770 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:52:53.314709 2584 scope.go:117] "RemoveContainer" containerID="a9e828b67c6d1482d22f98a1795eb4a443ceb054f1d36d7b587e1679a8fed3b5" Jul 17 09:52:53.314953 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:52:53.314928 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:53:06.993890 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:06.993853 2584 scope.go:117] "RemoveContainer" containerID="a9e828b67c6d1482d22f98a1795eb4a443ceb054f1d36d7b587e1679a8fed3b5" Jul 17 09:53:07.945831 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:07.945746 2584 generic.go:356] "Generic (PLEG): container finished" podID="3b692649-cf69-4039-8130-6f0e0e778702" containerID="bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63" exitCode=1 Jul 17 09:53:07.945831 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:07.945820 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerDied","Data":"bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63"} Jul 17 09:53:07.946058 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:07.945886 2584 scope.go:117] "RemoveContainer" containerID="a9e828b67c6d1482d22f98a1795eb4a443ceb054f1d36d7b587e1679a8fed3b5" Jul 17 09:53:07.946227 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:07.946206 2584 scope.go:117] "RemoveContainer" containerID="bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63" Jul 17 09:53:07.946477 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:53:07.946449 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:53:08.314587 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:08.314480 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:53:08.953787 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:08.953756 2584 scope.go:117] "RemoveContainer" containerID="bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63" Jul 17 09:53:08.954004 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:53:08.953983 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:53:13.314820 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:13.314779 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:53:13.315280 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:13.315215 2584 scope.go:117] "RemoveContainer" containerID="bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63" Jul 17 09:53:13.315441 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:53:13.315393 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:53:26.993291 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:26.993260 2584 scope.go:117] "RemoveContainer" containerID="bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63" Jul 17 09:53:26.993696 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:53:26.993499 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:53:40.993516 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:40.993481 2584 scope.go:117] "RemoveContainer" containerID="bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63" Jul 17 09:53:40.993922 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:53:40.993636 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:53:52.993967 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:52.993892 2584 scope.go:117] "RemoveContainer" containerID="bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63" Jul 17 09:53:54.188873 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:54.188841 2584 generic.go:356] "Generic (PLEG): container finished" podID="3b692649-cf69-4039-8130-6f0e0e778702" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" exitCode=1 Jul 17 09:53:54.189318 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:54.188889 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerDied","Data":"72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e"} Jul 17 09:53:54.189318 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:54.188934 2584 scope.go:117] "RemoveContainer" containerID="bf03c90f33e74ec963bec6a89493d62efa7996e6533980ba081a756c4344fb63" Jul 17 09:53:54.189318 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:54.189278 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:53:54.189541 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:53:54.189517 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:53:58.314698 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:58.314643 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:53:58.315114 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:53:58.315099 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:53:58.315296 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:53:58.315277 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:54:03.314289 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:54:03.314244 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:54:03.314988 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:54:03.314971 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:54:03.315190 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:54:03.315170 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:54:17.996663 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:54:17.996631 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:54:17.997175 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:54:17.996836 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:54:29.993481 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:54:29.993447 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:54:29.994098 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:54:29.993633 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:54:41.993908 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:54:41.993868 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:54:41.994300 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:54:41.994090 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:54:52.993385 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:54:52.993353 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:54:52.993821 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:54:52.993577 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:55:05.993514 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:05.993481 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:55:05.993989 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:05.993672 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:55:16.993556 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:16.993518 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:55:17.627330 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:17.627295 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerStarted","Data":"b7e809fa74734a7b5fa62e667ada88398ab57dbfbf12dc74f0ec471379970f9f"} Jul 17 09:55:17.627523 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:17.627509 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:55:17.649501 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:17.649441 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podStartSLOduration=173.233649543 podStartE2EDuration="2m55.649423102s" totalImagesPullingTime="2.415773559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:52:22 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:55:17.646631072 +0000 UTC m=+1700.185830050" watchObservedRunningTime="2026-07-17 09:55:17.649423102 +0000 UTC m=+1700.188622067" Jul 17 09:55:18.634502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:18.634468 2584 generic.go:356] "Generic (PLEG): container finished" podID="3b692649-cf69-4039-8130-6f0e0e778702" containerID="b7e809fa74734a7b5fa62e667ada88398ab57dbfbf12dc74f0ec471379970f9f" exitCode=1 Jul 17 09:55:18.634911 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:18.634514 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerDied","Data":"b7e809fa74734a7b5fa62e667ada88398ab57dbfbf12dc74f0ec471379970f9f"} Jul 17 09:55:18.634911 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:18.634565 2584 scope.go:117] "RemoveContainer" containerID="72d500b767e12ddcb0c91ff3f07d701e29d79be3acb8d38bb919ffb98127ce0e" Jul 17 09:55:18.634911 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:18.634783 2584 scope.go:117] "RemoveContainer" containerID="b7e809fa74734a7b5fa62e667ada88398ab57dbfbf12dc74f0ec471379970f9f" Jul 17 09:55:18.635020 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:18.634999 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:55:19.640674 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:19.640602 2584 scope.go:117] "RemoveContainer" containerID="b7e809fa74734a7b5fa62e667ada88398ab57dbfbf12dc74f0ec471379970f9f" Jul 17 09:55:19.641028 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:19.640785 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:55:23.314752 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:23.314722 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:55:23.315132 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:23.315090 2584 scope.go:117] "RemoveContainer" containerID="b7e809fa74734a7b5fa62e667ada88398ab57dbfbf12dc74f0ec471379970f9f" Jul 17 09:55:23.315267 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:23.315248 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-3fc19893-7d796947b9-92v7n_odh-ai-gateway-infra(3b692649-cf69-4039-8130-6f0e0e778702)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" podUID="3b692649-cf69-4039-8130-6f0e0e778702" Jul 17 09:55:28.366659 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:28.366623 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228"] Jul 17 09:55:28.367074 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:28.366815 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" containerID="cri-o://35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d" gracePeriod=30 Jul 17 09:55:33.608939 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.608908 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:55:33.715484 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.715452 2584 generic.go:356] "Generic (PLEG): container finished" podID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerID="35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d" exitCode=0 Jul 17 09:55:33.715635 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.715524 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" Jul 17 09:55:33.715635 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.715538 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" event={"ID":"b2e6aa51-f865-4401-ab5f-6ed4c1653a37","Type":"ContainerDied","Data":"35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d"} Jul 17 09:55:33.715635 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.715602 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228" event={"ID":"b2e6aa51-f865-4401-ab5f-6ed4c1653a37","Type":"ContainerDied","Data":"4825a9e062df90eccd2250403a0602a5aede2f944b499a9e4e67610d470320af"} Jul 17 09:55:33.715635 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.715626 2584 scope.go:117] "RemoveContainer" containerID="35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d" Jul 17 09:55:33.718304 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718269 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-socket\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718465 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718320 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-podinfo\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718465 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718358 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fblpt\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-kube-api-access-fblpt\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718465 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718388 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istiod-ca-cert\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718486 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-data\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718524 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-certs\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718549 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-envoy\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718580 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:55:33.718630 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718612 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-credential-socket\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718880 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718654 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-token\") pod \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\" (UID: \"b2e6aa51-f865-4401-ab5f-6ed4c1653a37\") " Jul 17 09:55:33.718933 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.718923 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.719708 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.719370 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 09:55:33.719708 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.719416 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:55:33.719708 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.719673 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:55:33.719959 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.719705 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:55:33.722842 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.722782 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-kube-api-access-fblpt" (OuterVolumeSpecName: "kube-api-access-fblpt") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "kube-api-access-fblpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:55:33.723877 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.723853 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:55:33.726229 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.726145 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 09:55:33.728001 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.727972 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2e6aa51-f865-4401-ab5f-6ed4c1653a37" (UID: "b2e6aa51-f865-4401-ab5f-6ed4c1653a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:55:33.729954 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.729932 2584 scope.go:117] "RemoveContainer" containerID="6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d" Jul 17 09:55:33.733120 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.733097 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n"] Jul 17 09:55:33.744688 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.744672 2584 scope.go:117] "RemoveContainer" containerID="35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d" Jul 17 09:55:33.744973 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:33.744957 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d\": container with ID starting with 35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d not found: ID does not exist" containerID="35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d" Jul 17 09:55:33.745018 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.744982 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d"} err="failed to get container status \"35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d\": rpc error: code = NotFound desc = could not find container \"35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d\": container with ID starting with 35ab05af61b1a33482da7986fefa15b83b5cd4e9cfda3c2a94de6438c045974d not found: ID does not exist" Jul 17 09:55:33.745018 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.744998 2584 scope.go:117] "RemoveContainer" containerID="6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d" Jul 17 09:55:33.745290 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:33.745274 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d\": container with ID starting with 6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d not found: ID does not exist" containerID="6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d" Jul 17 09:55:33.745358 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.745293 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d"} err="failed to get container status \"6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d\": rpc error: code = NotFound desc = could not find container \"6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d\": container with ID starting with 6ab0ec95d65a52b56cb485c3827f695070cebcce35b393ef0a3b016ed1641c1d not found: ID does not exist" Jul 17 09:55:33.820064 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.820027 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.820064 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.820052 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblpt\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-kube-api-access-fblpt\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.820064 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.820061 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.820064 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.820070 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.820371 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.820078 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.820371 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.820085 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.820371 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.820092 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.820371 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.820100 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e6aa51-f865-4401-ab5f-6ed4c1653a37-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:33.858236 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.858209 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:55:33.921045 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.920959 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls\") pod \"3b692649-cf69-4039-8130-6f0e0e778702\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " Jul 17 09:55:33.921045 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.921007 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b692649-cf69-4039-8130-6f0e0e778702-openshift-service-ca\") pod \"3b692649-cf69-4039-8130-6f0e0e778702\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " Jul 17 09:55:33.921284 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.921058 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvbj7\" (UniqueName: \"kubernetes.io/projected/3b692649-cf69-4039-8130-6f0e0e778702-kube-api-access-mvbj7\") pod \"3b692649-cf69-4039-8130-6f0e0e778702\" (UID: \"3b692649-cf69-4039-8130-6f0e0e778702\") " Jul 17 09:55:33.921431 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.921386 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b692649-cf69-4039-8130-6f0e0e778702-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b692649-cf69-4039-8130-6f0e0e778702" (UID: "3b692649-cf69-4039-8130-6f0e0e778702"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 09:55:33.923171 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.923149 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b692649-cf69-4039-8130-6f0e0e778702" (UID: "3b692649-cf69-4039-8130-6f0e0e778702"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 09:55:33.923297 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:33.923245 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b692649-cf69-4039-8130-6f0e0e778702-kube-api-access-mvbj7" (OuterVolumeSpecName: "kube-api-access-mvbj7") pod "3b692649-cf69-4039-8130-6f0e0e778702" (UID: "3b692649-cf69-4039-8130-6f0e0e778702"). InnerVolumeSpecName "kube-api-access-mvbj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:55:34.022027 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.022002 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b692649-cf69-4039-8130-6f0e0e778702-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:34.022027 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.022026 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvbj7\" (UniqueName: \"kubernetes.io/projected/3b692649-cf69-4039-8130-6f0e0e778702-kube-api-access-mvbj7\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:34.022154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.022035 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b692649-cf69-4039-8130-6f0e0e778702-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:55:34.031824 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.031800 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228"] Jul 17 09:55:34.036176 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.036150 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3fc19893-openshift-default-68f7994775-4c228"] Jul 17 09:55:34.723432 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.723331 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" Jul 17 09:55:34.723432 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.723359 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n" event={"ID":"3b692649-cf69-4039-8130-6f0e0e778702","Type":"ContainerDied","Data":"7dd03e5aad8577f4737b4832aca6928a09be9a62addb88f22f58ec620ffddbaf"} Jul 17 09:55:34.723432 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.723424 2584 scope.go:117] "RemoveContainer" containerID="b7e809fa74734a7b5fa62e667ada88398ab57dbfbf12dc74f0ec471379970f9f" Jul 17 09:55:34.740270 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.740243 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n"] Jul 17 09:55:34.743644 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:34.743624 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3fc19893-7d796947b9-92v7n"] Jul 17 09:55:35.999253 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:35.999221 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b692649-cf69-4039-8130-6f0e0e778702" path="/var/lib/kubelet/pods/3b692649-cf69-4039-8130-6f0e0e778702/volumes" Jul 17 09:55:35.999698 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:35.999685 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" path="/var/lib/kubelet/pods/b2e6aa51-f865-4401-ab5f-6ed4c1653a37/volumes" Jul 17 09:55:40.434766 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.434733 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8"] Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435136 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435149 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435162 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435167 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435173 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435178 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435186 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435191 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435198 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435203 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435210 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435209 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435215 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435579 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435275 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435579 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435283 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435579 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435289 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435579 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435294 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435579 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435301 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435579 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435308 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" Jul 17 09:55:40.435579 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435313 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:40.435579 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.435319 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e6aa51-f865-4401-ab5f-6ed4c1653a37" containerName="istio-proxy" Jul 17 09:55:40.440262 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.440233 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.443258 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.443236 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5109c499-gw-openshift-default-dockercfg-cw8hm\"" Jul 17 09:55:40.453854 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.453822 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8"] Jul 17 09:55:40.483833 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.483799 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae087d58-ed81-4596-9168-78cdf4b6b871-istiod-ca-cert\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.484020 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.483855 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-data\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.484020 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.483905 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8js8\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-kube-api-access-l8js8\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.484020 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.483985 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-podinfo\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.484154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.484062 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-credential-socket\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.484154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.484099 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-token\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.484221 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.484196 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-socket\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.484254 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.484221 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-certs\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.484254 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.484243 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-envoy\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.584885 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.584856 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-socket\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.584885 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.584888 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-certs\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585127 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.584906 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-envoy\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585127 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.584926 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae087d58-ed81-4596-9168-78cdf4b6b871-istiod-ca-cert\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585127 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585054 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-data\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585127 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585106 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8js8\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-kube-api-access-l8js8\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585369 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585174 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-podinfo\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585369 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585225 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-credential-socket\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585369 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585269 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-token\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585369 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585346 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-socket\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585597 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585495 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-data\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585597 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585553 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-certs\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585722 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585704 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-credential-socket\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.585779 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.585757 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae087d58-ed81-4596-9168-78cdf4b6b871-istiod-ca-cert\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.587251 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.587235 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-envoy\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.587646 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.587626 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-podinfo\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.594900 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.594876 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8js8\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-kube-api-access-l8js8\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.595008 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.594977 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-token\") pod \"e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.754538 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.754420 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:40.893555 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.893527 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8"] Jul 17 09:55:40.895150 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:55:40.895119 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae087d58_ed81_4596_9168_78cdf4b6b871.slice/crio-c767d5a94ec0ce43cef452e6a492b87d8008b7d351cd35928447e66048908e45 WatchSource:0}: Error finding container c767d5a94ec0ce43cef452e6a492b87d8008b7d351cd35928447e66048908e45: Status 404 returned error can't find the container with id c767d5a94ec0ce43cef452e6a492b87d8008b7d351cd35928447e66048908e45 Jul 17 09:55:40.897327 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.897295 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:55:40.897421 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.897360 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:55:40.897421 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:40.897390 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:55:41.772641 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.772605 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" event={"ID":"ae087d58-ed81-4596-9168-78cdf4b6b871","Type":"ContainerStarted","Data":"dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714"} Jul 17 09:55:41.773034 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.772663 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" event={"ID":"ae087d58-ed81-4596-9168-78cdf4b6b871","Type":"ContainerStarted","Data":"c767d5a94ec0ce43cef452e6a492b87d8008b7d351cd35928447e66048908e45"} Jul 17 09:55:41.796316 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.796254 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" podStartSLOduration=1.7962319089999998 podStartE2EDuration="1.796231909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:55:41.79421282 +0000 UTC m=+1724.333411786" watchObservedRunningTime="2026-07-17 09:55:41.796231909 +0000 UTC m=+1724.335430874" Jul 17 09:55:41.802890 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.802853 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb"] Jul 17 09:55:41.803495 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.803477 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:41.803541 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.803499 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:41.803541 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.803519 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:41.803541 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.803528 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b692649-cf69-4039-8130-6f0e0e778702" containerName="maas-api" Jul 17 09:55:41.807279 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.807255 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:41.809864 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.809838 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5109c499-serving-cert\"" Jul 17 09:55:41.812102 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.812074 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb"] Jul 17 09:55:41.899887 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.899851 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:41.900064 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.899946 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br29g\" (UniqueName: \"kubernetes.io/projected/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-kube-api-access-br29g\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:41.900064 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:41.900011 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.001332 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.001300 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.001519 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.001340 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br29g\" (UniqueName: \"kubernetes.io/projected/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-kube-api-access-br29g\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.001519 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.001375 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.001609 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:42.001532 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-serving-cert: secret "maas-api-e2e-migrate-5109c499-serving-cert" not found Jul 17 09:55:42.001609 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:42.001585 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls podName:0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2 nodeName:}" failed. No retries permitted until 2026-07-17 09:55:42.501569568 +0000 UTC m=+1725.040768512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls") pod "maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" (UID: "0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2") : secret "maas-api-e2e-migrate-5109c499-serving-cert" not found Jul 17 09:55:42.001968 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.001949 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-openshift-service-ca\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.013081 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.013063 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br29g\" (UniqueName: \"kubernetes.io/projected/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-kube-api-access-br29g\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.507385 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.507326 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.510768 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.510737 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls\") pod \"maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.720382 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.720345 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:42.755378 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.755351 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:42.760499 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.760443 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:42.781870 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.778744 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:42.781870 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.780192 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:42.890760 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:42.890735 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb"] Jul 17 09:55:42.892009 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:55:42.891987 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb6b4d0_9a53_4e4f_ab0e_9e2a4c1d6fc2.slice/crio-971d335af33b0d9d02363436841d80ac8fb25277b32326ed9f25352f10828d68 WatchSource:0}: Error finding container 971d335af33b0d9d02363436841d80ac8fb25277b32326ed9f25352f10828d68: Status 404 returned error can't find the container with id 971d335af33b0d9d02363436841d80ac8fb25277b32326ed9f25352f10828d68 Jul 17 09:55:43.784933 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:43.784901 2584 generic.go:356] "Generic (PLEG): container finished" podID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerID="9f291475a03599bf906a26d4d19678abdb830334012c33d7a382e42e8737d120" exitCode=1 Jul 17 09:55:43.785351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:43.784979 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" event={"ID":"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2","Type":"ContainerDied","Data":"9f291475a03599bf906a26d4d19678abdb830334012c33d7a382e42e8737d120"} Jul 17 09:55:43.785351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:43.785027 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" event={"ID":"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2","Type":"ContainerStarted","Data":"971d335af33b0d9d02363436841d80ac8fb25277b32326ed9f25352f10828d68"} Jul 17 09:55:43.785351 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:43.785254 2584 scope.go:117] "RemoveContainer" containerID="9f291475a03599bf906a26d4d19678abdb830334012c33d7a382e42e8737d120" Jul 17 09:55:44.792206 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:44.792126 2584 generic.go:356] "Generic (PLEG): container finished" podID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerID="7b75545de7feadcf622b4baaa6d8f573d5650d930da0e3a21e33748fa50344e5" exitCode=1 Jul 17 09:55:44.792206 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:44.792194 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" event={"ID":"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2","Type":"ContainerDied","Data":"7b75545de7feadcf622b4baaa6d8f573d5650d930da0e3a21e33748fa50344e5"} Jul 17 09:55:44.792684 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:44.792234 2584 scope.go:117] "RemoveContainer" containerID="9f291475a03599bf906a26d4d19678abdb830334012c33d7a382e42e8737d120" Jul 17 09:55:44.792684 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:44.792502 2584 scope.go:117] "RemoveContainer" containerID="7b75545de7feadcf622b4baaa6d8f573d5650d930da0e3a21e33748fa50344e5" Jul 17 09:55:44.792769 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:44.792711 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:55:45.798847 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:45.798821 2584 scope.go:117] "RemoveContainer" containerID="7b75545de7feadcf622b4baaa6d8f573d5650d930da0e3a21e33748fa50344e5" Jul 17 09:55:45.799222 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:45.799015 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:55:47.721307 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:47.721268 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:47.721743 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:47.721707 2584 scope.go:117] "RemoveContainer" containerID="7b75545de7feadcf622b4baaa6d8f573d5650d930da0e3a21e33748fa50344e5" Jul 17 09:55:47.721916 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:47.721898 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:55:52.721164 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:52.721115 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:55:52.721647 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:52.721549 2584 scope.go:117] "RemoveContainer" containerID="7b75545de7feadcf622b4baaa6d8f573d5650d930da0e3a21e33748fa50344e5" Jul 17 09:55:52.721735 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:55:52.721714 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:55:52.836848 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:52.836815 2584 generic.go:356] "Generic (PLEG): container finished" podID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerID="dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714" exitCode=0 Jul 17 09:55:52.837000 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:52.836891 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" event={"ID":"ae087d58-ed81-4596-9168-78cdf4b6b871","Type":"ContainerDied","Data":"dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714"} Jul 17 09:55:52.837363 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:52.837350 2584 scope.go:117] "RemoveContainer" containerID="dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714" Jul 17 09:55:52.838502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:52.838479 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:55:52.838562 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:52.838551 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:55:52.838597 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:52.838584 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:55:53.847453 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:53.847382 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" event={"ID":"ae087d58-ed81-4596-9168-78cdf4b6b871","Type":"ContainerStarted","Data":"2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3"} Jul 17 09:55:54.756764 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:54.756729 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:54.757108 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:54.757087 2584 patch_prober.go:28] interesting pod/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 17 09:55:54.757308 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:54.757279 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 17 09:55:55.755005 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:55.754965 2584 patch_prober.go:28] interesting pod/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 17 09:55:55.755585 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:55.755545 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 17 09:55:56.755503 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:56.755467 2584 patch_prober.go:28] interesting pod/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 17 09:55:56.756078 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:56.756043 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 17 09:55:57.758822 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:57.758791 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:57.759248 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:57.759183 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:55:57.759887 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:55:57.759869 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:56:05.994178 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:05.994146 2584 scope.go:117] "RemoveContainer" containerID="7b75545de7feadcf622b4baaa6d8f573d5650d930da0e3a21e33748fa50344e5" Jul 17 09:56:06.925028 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:06.924996 2584 generic.go:356] "Generic (PLEG): container finished" podID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerID="77fef33f5f06173f710303d952f6b8d235564497bec11df0a64d0b66afa3b051" exitCode=1 Jul 17 09:56:06.925287 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:06.925077 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" event={"ID":"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2","Type":"ContainerDied","Data":"77fef33f5f06173f710303d952f6b8d235564497bec11df0a64d0b66afa3b051"} Jul 17 09:56:06.925287 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:06.925153 2584 scope.go:117] "RemoveContainer" containerID="7b75545de7feadcf622b4baaa6d8f573d5650d930da0e3a21e33748fa50344e5" Jul 17 09:56:06.925515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:06.925500 2584 scope.go:117] "RemoveContainer" containerID="77fef33f5f06173f710303d952f6b8d235564497bec11df0a64d0b66afa3b051" Jul 17 09:56:06.925769 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:06.925750 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:56:07.721018 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:07.720982 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:56:07.931594 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:07.931561 2584 scope.go:117] "RemoveContainer" containerID="77fef33f5f06173f710303d952f6b8d235564497bec11df0a64d0b66afa3b051" Jul 17 09:56:07.931792 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:07.931773 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:56:12.721091 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:12.721058 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:56:12.721585 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:12.721468 2584 scope.go:117] "RemoveContainer" containerID="77fef33f5f06173f710303d952f6b8d235564497bec11df0a64d0b66afa3b051" Jul 17 09:56:12.721672 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:12.721651 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:56:23.993813 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:23.993775 2584 scope.go:117] "RemoveContainer" containerID="77fef33f5f06173f710303d952f6b8d235564497bec11df0a64d0b66afa3b051" Jul 17 09:56:23.994215 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:23.993966 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:56:34.993502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:34.993466 2584 scope.go:117] "RemoveContainer" containerID="77fef33f5f06173f710303d952f6b8d235564497bec11df0a64d0b66afa3b051" Jul 17 09:56:36.076621 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:36.076525 2584 generic.go:356] "Generic (PLEG): container finished" podID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerID="08723bbaf0ce07673dadc1c7d599984180bcc1b40fe54f6b4d63f5f9418de93f" exitCode=1 Jul 17 09:56:36.076621 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:36.076595 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" event={"ID":"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2","Type":"ContainerDied","Data":"08723bbaf0ce07673dadc1c7d599984180bcc1b40fe54f6b4d63f5f9418de93f"} Jul 17 09:56:36.077127 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:36.076655 2584 scope.go:117] "RemoveContainer" containerID="77fef33f5f06173f710303d952f6b8d235564497bec11df0a64d0b66afa3b051" Jul 17 09:56:36.077127 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:36.077033 2584 scope.go:117] "RemoveContainer" containerID="08723bbaf0ce07673dadc1c7d599984180bcc1b40fe54f6b4d63f5f9418de93f" Jul 17 09:56:36.077303 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:36.077275 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:56:37.720973 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:37.720937 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:56:37.721380 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:37.721338 2584 scope.go:117] "RemoveContainer" containerID="08723bbaf0ce07673dadc1c7d599984180bcc1b40fe54f6b4d63f5f9418de93f" Jul 17 09:56:37.721601 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:37.721580 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:56:42.720502 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:42.720469 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:56:42.720901 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:42.720880 2584 scope.go:117] "RemoveContainer" containerID="08723bbaf0ce07673dadc1c7d599984180bcc1b40fe54f6b4d63f5f9418de93f" Jul 17 09:56:42.721081 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:42.721060 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb_odh-ai-gateway-infra(0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" Jul 17 09:56:47.349455 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:47.349415 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8"] Jul 17 09:56:47.349934 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:47.349655 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" containerID="cri-o://2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3" gracePeriod=30 Jul 17 09:56:52.598406 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.598379 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:56:52.664387 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664358 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae087d58-ed81-4596-9168-78cdf4b6b871-istiod-ca-cert\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664553 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664421 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-token\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664553 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664491 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-credential-socket\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664553 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664532 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-envoy\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664712 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664583 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-socket\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664712 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664609 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-podinfo\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664712 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664643 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8js8\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-kube-api-access-l8js8\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664712 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664673 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-data\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664976 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664714 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-certs\") pod \"ae087d58-ed81-4596-9168-78cdf4b6b871\" (UID: \"ae087d58-ed81-4596-9168-78cdf4b6b871\") " Jul 17 09:56:52.664976 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664857 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:56:52.664976 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664905 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae087d58-ed81-4596-9168-78cdf4b6b871-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 09:56:52.664976 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.664905 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:56:52.665184 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.665115 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae087d58-ed81-4596-9168-78cdf4b6b871-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.665184 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.665138 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.665184 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.665152 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.665184 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.665114 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:56:52.665826 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.665796 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:56:52.666898 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.666868 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 09:56:52.667027 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.667009 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:56:52.667269 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.667246 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 09:56:52.667356 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.667275 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-kube-api-access-l8js8" (OuterVolumeSpecName: "kube-api-access-l8js8") pod "ae087d58-ed81-4596-9168-78cdf4b6b871" (UID: "ae087d58-ed81-4596-9168-78cdf4b6b871"). InnerVolumeSpecName "kube-api-access-l8js8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:56:52.765710 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.765681 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.765710 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.765704 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.765710 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.765714 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8js8\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-kube-api-access-l8js8\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.765921 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.765724 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.765921 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.765732 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae087d58-ed81-4596-9168-78cdf4b6b871-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.765921 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.765740 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae087d58-ed81-4596-9168-78cdf4b6b871-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:52.917015 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:52.916923 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb"] Jul 17 09:56:53.040191 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.040166 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:56:53.068076 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.068046 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-openshift-service-ca\") pod \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " Jul 17 09:56:53.068236 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.068125 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls\") pod \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " Jul 17 09:56:53.068236 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.068170 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br29g\" (UniqueName: \"kubernetes.io/projected/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-kube-api-access-br29g\") pod \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\" (UID: \"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2\") " Jul 17 09:56:53.068426 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.068392 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" (UID: "0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 09:56:53.068636 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.068619 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:53.070512 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.070486 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-kube-api-access-br29g" (OuterVolumeSpecName: "kube-api-access-br29g") pod "0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" (UID: "0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2"). InnerVolumeSpecName "kube-api-access-br29g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 09:56:53.070615 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.070509 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" (UID: "0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 09:56:53.167137 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.167059 2584 generic.go:356] "Generic (PLEG): container finished" podID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerID="2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3" exitCode=0 Jul 17 09:56:53.167137 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.167106 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" Jul 17 09:56:53.167336 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.167151 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" event={"ID":"ae087d58-ed81-4596-9168-78cdf4b6b871","Type":"ContainerDied","Data":"2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3"} Jul 17 09:56:53.167336 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.167212 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8" event={"ID":"ae087d58-ed81-4596-9168-78cdf4b6b871","Type":"ContainerDied","Data":"c767d5a94ec0ce43cef452e6a492b87d8008b7d351cd35928447e66048908e45"} Jul 17 09:56:53.167336 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.167242 2584 scope.go:117] "RemoveContainer" containerID="2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3" Jul 17 09:56:53.168590 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.168569 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" event={"ID":"0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2","Type":"ContainerDied","Data":"971d335af33b0d9d02363436841d80ac8fb25277b32326ed9f25352f10828d68"} Jul 17 09:56:53.168688 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.168608 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb" Jul 17 09:56:53.169173 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.169142 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:53.169248 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.169180 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br29g\" (UniqueName: \"kubernetes.io/projected/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2-kube-api-access-br29g\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 09:56:53.178154 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.178141 2584 scope.go:117] "RemoveContainer" containerID="dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714" Jul 17 09:56:53.186622 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.186606 2584 scope.go:117] "RemoveContainer" containerID="2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3" Jul 17 09:56:53.186876 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:53.186859 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3\": container with ID starting with 2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3 not found: ID does not exist" containerID="2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3" Jul 17 09:56:53.186927 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.186884 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3"} err="failed to get container status \"2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3\": rpc error: code = NotFound desc = could not find container \"2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3\": container with ID starting with 2cbf5584f390bcbb70b7cdb3556a541b943542d63aa6b2c610a5e39652cb22a3 not found: ID does not exist" Jul 17 09:56:53.186927 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.186901 2584 scope.go:117] "RemoveContainer" containerID="dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714" Jul 17 09:56:53.187134 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:56:53.187117 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714\": container with ID starting with dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714 not found: ID does not exist" containerID="dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714" Jul 17 09:56:53.187178 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.187139 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714"} err="failed to get container status \"dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714\": rpc error: code = NotFound desc = could not find container \"dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714\": container with ID starting with dc4850046fa3a9e80de281d547b3d5ba8345c105f979d2e7f3ef32418277a714 not found: ID does not exist" Jul 17 09:56:53.187178 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.187154 2584 scope.go:117] "RemoveContainer" containerID="08723bbaf0ce07673dadc1c7d599984180bcc1b40fe54f6b4d63f5f9418de93f" Jul 17 09:56:53.192392 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.192363 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8"] Jul 17 09:56:53.195617 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.195597 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5109c499-gw-openshift-default-7f6785f5f9-2v8z8"] Jul 17 09:56:53.205536 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.205510 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb"] Jul 17 09:56:53.209192 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.209169 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5109c499-84864f4f9b-lhlsb"] Jul 17 09:56:53.999532 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:53.999499 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" path="/var/lib/kubelet/pods/0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2/volumes" Jul 17 09:56:54.000103 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:54.000088 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" path="/var/lib/kubelet/pods/ae087d58-ed81-4596-9168-78cdf4b6b871/volumes" Jul 17 09:56:58.068521 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:58.068497 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:56:58.070465 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:58.070444 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 09:56:59.193365 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193342 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc"] Jul 17 09:56:59.193730 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193716 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" Jul 17 09:56:59.193730 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193730 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" Jul 17 09:56:59.193805 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193744 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:56:59.193805 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193749 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:56:59.193805 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193761 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:56:59.193805 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193766 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:56:59.193944 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193832 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:56:59.193944 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193840 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" Jul 17 09:56:59.193944 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193846 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" Jul 17 09:56:59.193944 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.193853 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:56:59.196728 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.196711 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.200318 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.200296 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-10ce7fe6-openshift-default-dockercfg-tw8nq\"" Jul 17 09:56:59.210984 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.210961 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc"] Jul 17 09:56:59.220244 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220212 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-envoy\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.220341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220258 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-data\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.220341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220275 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-token\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.220341 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220319 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cj2j\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-kube-api-access-6cj2j\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.220490 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220360 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-certs\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.220490 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220380 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-credential-socket\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.220490 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220420 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-socket\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.220490 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220461 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-podinfo\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.220490 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.220481 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istiod-ca-cert\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321260 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321226 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-certs\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321260 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321261 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-credential-socket\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321508 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321285 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-socket\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321508 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321308 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-podinfo\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321508 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321331 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istiod-ca-cert\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321508 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321370 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-envoy\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321725 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321545 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-data\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321725 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321588 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-token\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321725 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321627 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cj2j\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-kube-api-access-6cj2j\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321725 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321697 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-credential-socket\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321938 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321732 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-socket\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.321938 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.321857 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-data\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.322199 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.322170 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istiod-ca-cert\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.322302 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.322201 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-certs\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.323819 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.323794 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-envoy\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.324024 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.324006 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-podinfo\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.329899 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.329872 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-token\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.330026 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.330005 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cj2j\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-kube-api-access-6cj2j\") pod \"e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.507269 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.507184 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:56:59.670463 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.670441 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc"] Jul 17 09:56:59.671611 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:56:59.671586 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c1d42b_cbaa_486c_92e1_64e57b50dfdb.slice/crio-5254a7ae0bc4656ea897386259e74ff90b14f70007c8071beae47e404514c291 WatchSource:0}: Error finding container 5254a7ae0bc4656ea897386259e74ff90b14f70007c8071beae47e404514c291: Status 404 returned error can't find the container with id 5254a7ae0bc4656ea897386259e74ff90b14f70007c8071beae47e404514c291 Jul 17 09:56:59.674181 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.674153 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:56:59.674274 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.674211 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:56:59.674274 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:56:59.674237 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:57:00.208190 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.208159 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" event={"ID":"30c1d42b-cbaa-486c-92e1-64e57b50dfdb","Type":"ContainerStarted","Data":"76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa"} Jul 17 09:57:00.208606 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.208207 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" event={"ID":"30c1d42b-cbaa-486c-92e1-64e57b50dfdb","Type":"ContainerStarted","Data":"5254a7ae0bc4656ea897386259e74ff90b14f70007c8071beae47e404514c291"} Jul 17 09:57:00.248104 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.248061 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" podStartSLOduration=1.248047179 podStartE2EDuration="1.248047179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:57:00.247745848 +0000 UTC m=+1802.786944814" watchObservedRunningTime="2026-07-17 09:57:00.248047179 +0000 UTC m=+1802.787246191" Jul 17 09:57:00.507746 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.507656 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:57:00.510974 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.510948 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns"] Jul 17 09:57:00.511349 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.511335 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" Jul 17 09:57:00.511349 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.511351 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae087d58-ed81-4596-9168-78cdf4b6b871" containerName="istio-proxy" Jul 17 09:57:00.511515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.511364 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:57:00.511515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.511369 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:57:00.511515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.511381 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:57:00.511515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.511387 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:57:00.511515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.511464 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:57:00.511515 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.511475 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cb6b4d0-9a53-4e4f-ab0e-9e2a4c1d6fc2" containerName="maas-api" Jul 17 09:57:00.514547 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.514530 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:57:00.514640 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.514620 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.517007 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.516986 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-10ce7fe6-serving-cert\"" Jul 17 09:57:00.523642 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.523619 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns"] Jul 17 09:57:00.634914 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.634878 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6a50d0-c988-4336-8068-03e945b27d20-openshift-service-ca\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.634914 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.634918 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zln6s\" (UniqueName: \"kubernetes.io/projected/df6a50d0-c988-4336-8068-03e945b27d20-kube-api-access-zln6s\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.635125 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.634944 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6a50d0-c988-4336-8068-03e945b27d20-maas-api-tls\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.736014 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.735970 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6a50d0-c988-4336-8068-03e945b27d20-openshift-service-ca\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.736014 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.736018 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zln6s\" (UniqueName: \"kubernetes.io/projected/df6a50d0-c988-4336-8068-03e945b27d20-kube-api-access-zln6s\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.736204 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.736038 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6a50d0-c988-4336-8068-03e945b27d20-maas-api-tls\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.736745 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.736718 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6a50d0-c988-4336-8068-03e945b27d20-openshift-service-ca\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.738491 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.738468 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6a50d0-c988-4336-8068-03e945b27d20-maas-api-tls\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.743602 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.743579 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zln6s\" (UniqueName: \"kubernetes.io/projected/df6a50d0-c988-4336-8068-03e945b27d20-kube-api-access-zln6s\") pod \"maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.827888 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.827819 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:00.957417 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:00.957361 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns"] Jul 17 09:57:00.961455 ip-10-0-133-236 kubenswrapper[2584]: W0717 09:57:00.961429 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf6a50d0_c988_4336_8068_03e945b27d20.slice/crio-74ab970f423ba992a0d2b31ebd59a7757d059c02f006fd6990a9ae05a130426f WatchSource:0}: Error finding container 74ab970f423ba992a0d2b31ebd59a7757d059c02f006fd6990a9ae05a130426f: Status 404 returned error can't find the container with id 74ab970f423ba992a0d2b31ebd59a7757d059c02f006fd6990a9ae05a130426f Jul 17 09:57:01.214371 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:01.214339 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerStarted","Data":"74ab970f423ba992a0d2b31ebd59a7757d059c02f006fd6990a9ae05a130426f"} Jul 17 09:57:01.214803 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:01.214645 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:57:01.215617 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:01.215601 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:57:02.221128 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:02.221089 2584 generic.go:356] "Generic (PLEG): container finished" podID="df6a50d0-c988-4336-8068-03e945b27d20" containerID="3d89a6de532348cc2f3b81e7be5b20e3384bc17dbb8255782e7067ef90d92741" exitCode=1 Jul 17 09:57:02.221517 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:02.221172 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerDied","Data":"3d89a6de532348cc2f3b81e7be5b20e3384bc17dbb8255782e7067ef90d92741"} Jul 17 09:57:02.221517 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:02.221436 2584 scope.go:117] "RemoveContainer" containerID="3d89a6de532348cc2f3b81e7be5b20e3384bc17dbb8255782e7067ef90d92741" Jul 17 09:57:03.229648 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:03.229569 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerStarted","Data":"f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0"} Jul 17 09:57:03.229987 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:03.229651 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:03.249328 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:03.249284 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podStartSLOduration=2.148367562 podStartE2EDuration="3.249270328s" totalImagesPullingTime="1.100902766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 09:57:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 09:57:03.247426672 +0000 UTC m=+1805.786625635" watchObservedRunningTime="2026-07-17 09:57:03.249270328 +0000 UTC m=+1805.788469294" Jul 17 09:57:04.238853 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:04.238821 2584 generic.go:356] "Generic (PLEG): container finished" podID="df6a50d0-c988-4336-8068-03e945b27d20" containerID="f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0" exitCode=1 Jul 17 09:57:04.239313 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:04.238881 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerDied","Data":"f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0"} Jul 17 09:57:04.239313 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:04.238940 2584 scope.go:117] "RemoveContainer" containerID="3d89a6de532348cc2f3b81e7be5b20e3384bc17dbb8255782e7067ef90d92741" Jul 17 09:57:04.239313 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:04.239232 2584 scope.go:117] "RemoveContainer" containerID="f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0" Jul 17 09:57:04.239508 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:04.239477 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:05.248881 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:05.248848 2584 scope.go:117] "RemoveContainer" containerID="f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0" Jul 17 09:57:05.249359 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:05.249096 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:08.229648 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:08.229612 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:08.230121 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:08.230075 2584 scope.go:117] "RemoveContainer" containerID="f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0" Jul 17 09:57:08.230310 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:08.230286 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:10.828187 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:10.828141 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:10.828726 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:10.828707 2584 scope.go:117] "RemoveContainer" containerID="f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0" Jul 17 09:57:10.829004 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:10.828978 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:13.297199 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:13.297165 2584 generic.go:356] "Generic (PLEG): container finished" podID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerID="76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa" exitCode=0 Jul 17 09:57:13.297613 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:13.297243 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" event={"ID":"30c1d42b-cbaa-486c-92e1-64e57b50dfdb","Type":"ContainerDied","Data":"76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa"} Jul 17 09:57:13.297679 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:13.297665 2584 scope.go:117] "RemoveContainer" containerID="76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa" Jul 17 09:57:13.298627 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:13.298603 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:57:13.298685 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:13.298666 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:57:13.298720 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:13.298700 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 09:57:14.303840 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:14.303805 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" event={"ID":"30c1d42b-cbaa-486c-92e1-64e57b50dfdb","Type":"ContainerStarted","Data":"8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa"} Jul 17 09:57:14.507560 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:14.507530 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:57:14.508343 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:14.508314 2584 patch_prober.go:28] interesting pod/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 17 09:57:14.508480 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:14.508368 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 17 09:57:15.508181 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:15.508147 2584 patch_prober.go:28] interesting pod/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 17 09:57:15.508591 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:15.508210 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 17 09:57:16.215462 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:16.215431 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:57:16.507943 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:16.507863 2584 patch_prober.go:28] interesting pod/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 17 09:57:16.507943 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:16.507915 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 17 09:57:17.508317 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:17.508282 2584 patch_prober.go:28] interesting pod/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 17 09:57:17.508731 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:17.508352 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 17 09:57:18.511034 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:18.510996 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:57:18.511976 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:18.511959 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 09:57:23.993905 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:23.993873 2584 scope.go:117] "RemoveContainer" containerID="f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0" Jul 17 09:57:23.994281 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:23.994153 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 09:57:25.365920 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:25.365893 2584 generic.go:356] "Generic (PLEG): container finished" podID="df6a50d0-c988-4336-8068-03e945b27d20" containerID="5572ff2f538b7cb4a0a1e8ddb468bd5a61a4cd3d02ff3f0126eebcdc3ba9e9ac" exitCode=1 Jul 17 09:57:25.366311 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:25.365938 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerDied","Data":"5572ff2f538b7cb4a0a1e8ddb468bd5a61a4cd3d02ff3f0126eebcdc3ba9e9ac"} Jul 17 09:57:25.366311 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:25.365986 2584 scope.go:117] "RemoveContainer" containerID="f2bed645ea4dc56a2681bab31370abf75e26778e2cfb000f2a6ba6cf0c9dc1a0" Jul 17 09:57:25.366419 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:25.366313 2584 scope.go:117] "RemoveContainer" containerID="5572ff2f538b7cb4a0a1e8ddb468bd5a61a4cd3d02ff3f0126eebcdc3ba9e9ac" Jul 17 09:57:25.366558 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:25.366535 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:28.230297 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:28.230269 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:28.230748 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:28.230645 2584 scope.go:117] "RemoveContainer" containerID="5572ff2f538b7cb4a0a1e8ddb468bd5a61a4cd3d02ff3f0126eebcdc3ba9e9ac" Jul 17 09:57:28.230826 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:28.230807 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:30.828163 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:30.828128 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:30.828601 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:30.828553 2584 scope.go:117] "RemoveContainer" containerID="5572ff2f538b7cb4a0a1e8ddb468bd5a61a4cd3d02ff3f0126eebcdc3ba9e9ac" Jul 17 09:57:30.828740 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:30.828719 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:41.993485 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:41.993447 2584 scope.go:117] "RemoveContainer" containerID="5572ff2f538b7cb4a0a1e8ddb468bd5a61a4cd3d02ff3f0126eebcdc3ba9e9ac" Jul 17 09:57:41.993889 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:41.993703 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:54.993780 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:54.993747 2584 scope.go:117] "RemoveContainer" containerID="5572ff2f538b7cb4a0a1e8ddb468bd5a61a4cd3d02ff3f0126eebcdc3ba9e9ac" Jul 17 09:57:55.523849 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:55.523819 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerStarted","Data":"d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa"} Jul 17 09:57:55.524035 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:55.524015 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:57:56.530871 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:56.530838 2584 generic.go:356] "Generic (PLEG): container finished" podID="df6a50d0-c988-4336-8068-03e945b27d20" containerID="d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa" exitCode=1 Jul 17 09:57:56.531354 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:56.530886 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerDied","Data":"d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa"} Jul 17 09:57:56.531354 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:56.530932 2584 scope.go:117] "RemoveContainer" containerID="5572ff2f538b7cb4a0a1e8ddb468bd5a61a4cd3d02ff3f0126eebcdc3ba9e9ac" Jul 17 09:57:56.531354 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:56.531206 2584 scope.go:117] "RemoveContainer" containerID="d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa" Jul 17 09:57:56.531547 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:56.531507 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:57:57.538587 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:57:57.538558 2584 scope.go:117] "RemoveContainer" containerID="d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa" Jul 17 09:57:57.538965 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:57:57.538753 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:58:00.828877 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:00.828841 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:58:00.829285 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:00.829232 2584 scope.go:117] "RemoveContainer" containerID="d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa" Jul 17 09:58:00.829473 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:58:00.829451 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:58:15.994119 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:15.994092 2584 scope.go:117] "RemoveContainer" containerID="d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa" Jul 17 09:58:15.994522 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:58:15.994282 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:58:29.994027 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:29.993949 2584 scope.go:117] "RemoveContainer" containerID="d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa" Jul 17 09:58:29.994500 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:58:29.994130 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:58:44.993892 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:44.993862 2584 scope.go:117] "RemoveContainer" containerID="d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa" Jul 17 09:58:45.787546 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:45.787514 2584 generic.go:356] "Generic (PLEG): container finished" podID="df6a50d0-c988-4336-8068-03e945b27d20" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" exitCode=1 Jul 17 09:58:45.787716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:45.787553 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerDied","Data":"d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58"} Jul 17 09:58:45.787716 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:45.787605 2584 scope.go:117] "RemoveContainer" containerID="d5efc9dabaff0d1271ba0e0555e92ae7605c70cb205cbdbc75fa806f381f9eaa" Jul 17 09:58:45.792415 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:45.788164 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 09:58:45.792415 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:58:45.788830 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:58:50.524347 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:50.524314 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:58:50.524761 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:50.524744 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 09:58:50.524954 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:58:50.524933 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:58:50.828691 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:50.828606 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 09:58:50.828996 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:58:50.828984 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 09:58:50.829173 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:58:50.829154 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:59:01.994088 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:59:01.994051 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 09:59:01.994595 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:59:01.994208 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:59:16.993140 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:59:16.993105 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 09:59:16.993568 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:59:16.993310 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:59:28.993577 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:59:28.993545 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 09:59:28.993966 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:59:28.993751 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:59:39.993976 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:59:39.993938 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 09:59:39.994642 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:59:39.994124 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 09:59:52.993276 ip-10-0-133-236 kubenswrapper[2584]: I0717 09:59:52.993196 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 09:59:52.993689 ip-10-0-133-236 kubenswrapper[2584]: E0717 09:59:52.993362 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:00:05.993134 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:05.993091 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 10:00:07.224836 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:07.224805 2584 generic.go:356] "Generic (PLEG): container finished" podID="df6a50d0-c988-4336-8068-03e945b27d20" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" exitCode=1 Jul 17 10:00:07.225259 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:07.224861 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerDied","Data":"b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25"} Jul 17 10:00:07.225259 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:07.224907 2584 scope.go:117] "RemoveContainer" containerID="d5ba336a42095309bd7756ee5da6b7a85cfd08c0c6a23766ff13750d9ebc2c58" Jul 17 10:00:07.225259 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:07.225178 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:00:07.225436 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:00:07.225390 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:00:10.524957 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:10.524927 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 10:00:10.525364 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:10.525271 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:00:10.525512 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:00:10.525491 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:00:10.828065 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:10.827992 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 10:00:11.246660 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:11.246633 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:00:11.246826 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:00:11.246797 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:00:26.993935 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:26.993906 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:00:26.994317 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:00:26.994082 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:00:37.998024 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:37.997997 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:00:37.998514 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:00:37.998163 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:00:50.993561 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:00:50.993529 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:00:50.994058 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:00:50.993753 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:01:02.993323 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:01:02.993292 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:01:02.993732 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:01:02.993499 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:01:16.993510 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:01:16.993478 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:01:16.993873 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:01:16.993648 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:01:31.993440 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:01:31.993370 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:01:31.993849 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:01:31.993552 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:01:42.993961 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:01:42.993927 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:01:42.994441 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:01:42.994137 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:01:55.999147 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:01:55.999119 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:01:55.999535 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:01:55.999314 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:01:58.105407 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:01:58.105364 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:01:58.107332 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:01:58.107313 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:02:10.993101 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:10.993073 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:02:10.993638 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:02:10.993280 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:02:22.993491 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:22.993460 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:02:22.995776 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:02:22.993618 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:02:36.993752 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:36.993718 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:02:36.994136 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:02:36.993924 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:02:50.993837 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:50.993761 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:02:50.994304 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:50.994084 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 10:02:52.078773 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:52.078742 2584 generic.go:356] "Generic (PLEG): container finished" podID="df6a50d0-c988-4336-8068-03e945b27d20" containerID="c526b91136326a8909c6d20c424dde9f531f57fc3218b8ec506cefd3e9758325" exitCode=1 Jul 17 10:02:52.079164 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:52.078809 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerDied","Data":"c526b91136326a8909c6d20c424dde9f531f57fc3218b8ec506cefd3e9758325"} Jul 17 10:02:52.079164 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:52.078869 2584 scope.go:117] "RemoveContainer" containerID="b683e83ea75b2ce3c02501ce0b3e25c43359e18ced58ac83a17020ca720ccb25" Jul 17 10:02:52.079291 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:52.079192 2584 scope.go:117] "RemoveContainer" containerID="c526b91136326a8909c6d20c424dde9f531f57fc3218b8ec506cefd3e9758325" Jul 17 10:02:52.079442 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:02:52.079418 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:02:55.524737 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:55.524698 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 10:02:55.525133 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:02:55.525079 2584 scope.go:117] "RemoveContainer" containerID="c526b91136326a8909c6d20c424dde9f531f57fc3218b8ec506cefd3e9758325" Jul 17 10:02:55.525285 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:02:55.525261 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:03:00.828358 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:00.828321 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 10:03:00.828806 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:00.828791 2584 scope.go:117] "RemoveContainer" containerID="c526b91136326a8909c6d20c424dde9f531f57fc3218b8ec506cefd3e9758325" Jul 17 10:03:00.828987 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:00.828968 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:03:07.712504 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:07.712468 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc"] Jul 17 10:03:07.712991 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:07.712691 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" containerID="cri-o://8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa" gracePeriod=30 Jul 17 10:03:11.993958 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:11.993922 2584 scope.go:117] "RemoveContainer" containerID="c526b91136326a8909c6d20c424dde9f531f57fc3218b8ec506cefd3e9758325" Jul 17 10:03:11.994495 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:11.994093 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns_odh-ai-gateway-infra(df6a50d0-c988-4336-8068-03e945b27d20)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" podUID="df6a50d0-c988-4336-8068-03e945b27d20" Jul 17 10:03:12.961270 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:12.961243 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 10:03:13.061547 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061498 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-socket\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.061547 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061552 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-envoy\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.062014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061582 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cj2j\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-kube-api-access-6cj2j\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.062014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061676 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-podinfo\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.062014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061705 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-data\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.062014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061731 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-certs\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.062014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061771 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-credential-socket\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.062014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061801 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istiod-ca-cert\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.062014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061864 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-token\") pod \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\" (UID: \"30c1d42b-cbaa-486c-92e1-64e57b50dfdb\") " Jul 17 10:03:13.062014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.061826 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:03:13.062477 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.062079 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:03:13.062477 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.062121 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:03:13.063197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.062607 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.063197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.062634 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.063197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.062650 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.063197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.062717 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:03:13.063539 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.063427 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:03:13.064075 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.064047 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:03:13.064075 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.064062 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-kube-api-access-6cj2j" (OuterVolumeSpecName: "kube-api-access-6cj2j") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "kube-api-access-6cj2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:03:13.064207 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.064096 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 10:03:13.064207 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.064133 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "30c1d42b-cbaa-486c-92e1-64e57b50dfdb" (UID: "30c1d42b-cbaa-486c-92e1-64e57b50dfdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:03:13.163467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.163431 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.163467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.163459 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cj2j\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-kube-api-access-6cj2j\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.163467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.163469 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.163467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.163479 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.163742 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.163488 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.163742 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.163495 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c1d42b-cbaa-486c-92e1-64e57b50dfdb-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:13.194175 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.194144 2584 generic.go:356] "Generic (PLEG): container finished" podID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerID="8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa" exitCode=0 Jul 17 10:03:13.194329 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.194190 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" event={"ID":"30c1d42b-cbaa-486c-92e1-64e57b50dfdb","Type":"ContainerDied","Data":"8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa"} Jul 17 10:03:13.194329 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.194210 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" Jul 17 10:03:13.194329 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.194249 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc" event={"ID":"30c1d42b-cbaa-486c-92e1-64e57b50dfdb","Type":"ContainerDied","Data":"5254a7ae0bc4656ea897386259e74ff90b14f70007c8071beae47e404514c291"} Jul 17 10:03:13.194329 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.194275 2584 scope.go:117] "RemoveContainer" containerID="8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa" Jul 17 10:03:13.204410 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.204370 2584 scope.go:117] "RemoveContainer" containerID="76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa" Jul 17 10:03:13.212434 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.212417 2584 scope.go:117] "RemoveContainer" containerID="8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa" Jul 17 10:03:13.216115 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:13.216083 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa\": container with ID starting with 8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa not found: ID does not exist" containerID="8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa" Jul 17 10:03:13.216235 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.216124 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa"} err="failed to get container status \"8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa\": rpc error: code = NotFound desc = could not find container \"8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa\": container with ID starting with 8a30224901d95097b8491d46f8130911b3075f6fb2b4c2d0884554c17fe7b0fa not found: ID does not exist" Jul 17 10:03:13.216235 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.216146 2584 scope.go:117] "RemoveContainer" containerID="76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa" Jul 17 10:03:13.216686 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:13.216663 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa\": container with ID starting with 76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa not found: ID does not exist" containerID="76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa" Jul 17 10:03:13.216735 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.216697 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa"} err="failed to get container status \"76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa\": rpc error: code = NotFound desc = could not find container \"76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa\": container with ID starting with 76c0bebde692b30db0cafbf3b1fc8ae14882df1ba5cb7a2125a828c000a865aa not found: ID does not exist" Jul 17 10:03:13.218812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.218785 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc"] Jul 17 10:03:13.222038 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.222018 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-10ce7fe6-openshift-default-68c7f9bb77-krdxc"] Jul 17 10:03:13.851322 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.851287 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns"] Jul 17 10:03:13.979781 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:13.979744 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 10:03:14.001507 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.001475 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" path="/var/lib/kubelet/pods/30c1d42b-cbaa-486c-92e1-64e57b50dfdb/volumes" Jul 17 10:03:14.071200 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.071163 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zln6s\" (UniqueName: \"kubernetes.io/projected/df6a50d0-c988-4336-8068-03e945b27d20-kube-api-access-zln6s\") pod \"df6a50d0-c988-4336-8068-03e945b27d20\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " Jul 17 10:03:14.071622 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.071244 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6a50d0-c988-4336-8068-03e945b27d20-openshift-service-ca\") pod \"df6a50d0-c988-4336-8068-03e945b27d20\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " Jul 17 10:03:14.071622 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.071366 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6a50d0-c988-4336-8068-03e945b27d20-maas-api-tls\") pod \"df6a50d0-c988-4336-8068-03e945b27d20\" (UID: \"df6a50d0-c988-4336-8068-03e945b27d20\") " Jul 17 10:03:14.071622 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.071605 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df6a50d0-c988-4336-8068-03e945b27d20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df6a50d0-c988-4336-8068-03e945b27d20" (UID: "df6a50d0-c988-4336-8068-03e945b27d20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:03:14.073342 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.073319 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6a50d0-c988-4336-8068-03e945b27d20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df6a50d0-c988-4336-8068-03e945b27d20" (UID: "df6a50d0-c988-4336-8068-03e945b27d20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:03:14.073467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.073355 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6a50d0-c988-4336-8068-03e945b27d20-kube-api-access-zln6s" (OuterVolumeSpecName: "kube-api-access-zln6s") pod "df6a50d0-c988-4336-8068-03e945b27d20" (UID: "df6a50d0-c988-4336-8068-03e945b27d20"). InnerVolumeSpecName "kube-api-access-zln6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:03:14.172507 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.172424 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df6a50d0-c988-4336-8068-03e945b27d20-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:14.172507 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.172452 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df6a50d0-c988-4336-8068-03e945b27d20-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:14.172507 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.172462 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zln6s\" (UniqueName: \"kubernetes.io/projected/df6a50d0-c988-4336-8068-03e945b27d20-kube-api-access-zln6s\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:03:14.201650 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.201619 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" Jul 17 10:03:14.201835 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.201664 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns" event={"ID":"df6a50d0-c988-4336-8068-03e945b27d20","Type":"ContainerDied","Data":"74ab970f423ba992a0d2b31ebd59a7757d059c02f006fd6990a9ae05a130426f"} Jul 17 10:03:14.201835 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.201723 2584 scope.go:117] "RemoveContainer" containerID="c526b91136326a8909c6d20c424dde9f531f57fc3218b8ec506cefd3e9758325" Jul 17 10:03:14.224746 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.224720 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns"] Jul 17 10:03:14.227093 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:14.227070 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-10ce7fe6-8667b75d46-zspns"] Jul 17 10:03:15.999517 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:15.999487 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df6a50d0-c988-4336-8068-03e945b27d20" path="/var/lib/kubelet/pods/df6a50d0-c988-4336-8068-03e945b27d20/volumes" Jul 17 10:03:19.626771 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.626731 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7"] Jul 17 10:03:19.627317 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627297 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627362 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627322 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627362 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627344 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627362 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627352 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627505 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627364 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627505 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627374 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627505 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627417 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" Jul 17 10:03:19.627505 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627426 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" Jul 17 10:03:19.627505 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627436 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627505 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627444 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627505 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627469 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" Jul 17 10:03:19.627505 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627480 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" Jul 17 10:03:19.627738 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627577 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627738 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627586 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627738 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627594 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" Jul 17 10:03:19.627738 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627603 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c1d42b-cbaa-486c-92e1-64e57b50dfdb" containerName="istio-proxy" Jul 17 10:03:19.627738 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627612 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627738 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627621 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627738 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627629 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.627738 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.627640 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:19.632534 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.632243 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.634945 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.634926 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-125842b9-openshift-default-dockercfg-9bcph\"" Jul 17 10:03:19.639770 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.639745 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7"] Jul 17 10:03:19.720766 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.720727 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40223604-833e-43ef-85df-1873ee58eadc-istio-podinfo\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.720929 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.720774 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-istio-token\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.720929 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.720859 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-data\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.720929 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.720894 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rvr\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-kube-api-access-d8rvr\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.720929 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.720926 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-credential-socket\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.721085 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.720950 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-envoy\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.721085 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.720975 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-socket\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.721085 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.721026 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-certs\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.721085 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.721047 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40223604-833e-43ef-85df-1873ee58eadc-istiod-ca-cert\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822278 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822217 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-socket\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822486 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822308 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-certs\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822486 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822339 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40223604-833e-43ef-85df-1873ee58eadc-istiod-ca-cert\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822486 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822441 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40223604-833e-43ef-85df-1873ee58eadc-istio-podinfo\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822486 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822478 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-istio-token\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822698 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822533 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-data\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822698 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822579 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rvr\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-kube-api-access-d8rvr\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822797 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822747 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-socket\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822847 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822813 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-credential-socket\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822847 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822830 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-certs\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822948 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822857 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-envoy\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.822948 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.822902 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-data\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.823185 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.823162 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40223604-833e-43ef-85df-1873ee58eadc-istiod-ca-cert\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.823248 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.823196 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-credential-socket\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.824880 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.824856 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-envoy\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.825034 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.825015 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40223604-833e-43ef-85df-1873ee58eadc-istio-podinfo\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.831907 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.831881 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-istio-token\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.832067 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.832002 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rvr\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-kube-api-access-d8rvr\") pod \"e2e-derive-125842b9-openshift-default-677df58467-csvx7\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:19.945649 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:19.945556 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:20.084742 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.084716 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7"] Jul 17 10:03:20.086670 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:03:20.086636 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40223604_833e_43ef_85df_1873ee58eadc.slice/crio-fa6c709d2e22aad1d45db0f3fd2631b22573da20e02cb37498929973a29d15e5 WatchSource:0}: Error finding container fa6c709d2e22aad1d45db0f3fd2631b22573da20e02cb37498929973a29d15e5: Status 404 returned error can't find the container with id fa6c709d2e22aad1d45db0f3fd2631b22573da20e02cb37498929973a29d15e5 Jul 17 10:03:20.088779 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.088750 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:03:20.088863 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.088810 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:03:20.088863 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.088838 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:03:20.237085 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.237005 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" event={"ID":"40223604-833e-43ef-85df-1873ee58eadc","Type":"ContainerStarted","Data":"fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f"} Jul 17 10:03:20.237085 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.237058 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" event={"ID":"40223604-833e-43ef-85df-1873ee58eadc","Type":"ContainerStarted","Data":"fa6c709d2e22aad1d45db0f3fd2631b22573da20e02cb37498929973a29d15e5"} Jul 17 10:03:20.261810 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.261755 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" podStartSLOduration=1.261738736 podStartE2EDuration="1.261738736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:03:20.26062334 +0000 UTC m=+2182.799822307" watchObservedRunningTime="2026-07-17 10:03:20.261738736 +0000 UTC m=+2182.800937701" Jul 17 10:03:20.892145 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.892113 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb"] Jul 17 10:03:20.892686 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.892667 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:20.892801 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.892689 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:20.892801 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.892709 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:20.892801 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.892718 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:20.892801 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.892730 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:20.892801 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.892739 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:20.893074 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.892854 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6a50d0-c988-4336-8068-03e945b27d20" containerName="maas-api" Jul 17 10:03:20.895978 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.895958 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:20.899420 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.899384 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-125842b9-serving-cert\"" Jul 17 10:03:20.910886 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.910861 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb"] Jul 17 10:03:20.946544 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:20.946519 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:21.032293 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.032254 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e141fd-cd0a-4b37-a576-f318fee3308d-openshift-service-ca\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.032545 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.032305 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.032545 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.032332 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mhr2\" (UniqueName: \"kubernetes.io/projected/79e141fd-cd0a-4b37-a576-f318fee3308d-kube-api-access-6mhr2\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.133584 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.133544 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.133770 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.133592 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mhr2\" (UniqueName: \"kubernetes.io/projected/79e141fd-cd0a-4b37-a576-f318fee3308d-kube-api-access-6mhr2\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.133770 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:21.133698 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-serving-cert: secret "maas-api-e2e-derive-125842b9-serving-cert" not found Jul 17 10:03:21.133883 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:21.133775 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls podName:79e141fd-cd0a-4b37-a576-f318fee3308d nodeName:}" failed. No retries permitted until 2026-07-17 10:03:21.63375726 +0000 UTC m=+2184.172956204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls") pod "maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" (UID: "79e141fd-cd0a-4b37-a576-f318fee3308d") : secret "maas-api-e2e-derive-125842b9-serving-cert" not found Jul 17 10:03:21.133883 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.133700 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e141fd-cd0a-4b37-a576-f318fee3308d-openshift-service-ca\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.134328 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.134306 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e141fd-cd0a-4b37-a576-f318fee3308d-openshift-service-ca\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.150003 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.149948 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mhr2\" (UniqueName: \"kubernetes.io/projected/79e141fd-cd0a-4b37-a576-f318fee3308d-kube-api-access-6mhr2\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.639579 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.639543 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.641959 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.641937 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls\") pod \"maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.806448 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.806390 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:21.943530 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.943504 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb"] Jul 17 10:03:21.945301 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:03:21.945273 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e141fd_cd0a_4b37_a576_f318fee3308d.slice/crio-61f391fa3865a6aa7c3ff6cd56c27b7846944002259ddd79a7f3fa047e828b44 WatchSource:0}: Error finding container 61f391fa3865a6aa7c3ff6cd56c27b7846944002259ddd79a7f3fa047e828b44: Status 404 returned error can't find the container with id 61f391fa3865a6aa7c3ff6cd56c27b7846944002259ddd79a7f3fa047e828b44 Jul 17 10:03:21.951090 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:21.951066 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:22.247274 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:22.247183 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerStarted","Data":"61f391fa3865a6aa7c3ff6cd56c27b7846944002259ddd79a7f3fa047e828b44"} Jul 17 10:03:22.247453 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:22.247432 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:22.248497 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:22.248479 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:23.253853 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:23.253824 2584 generic.go:356] "Generic (PLEG): container finished" podID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerID="952cc8767575ee677829afbc8e3e9941cf78c898eaff728b2b28fcf86e2c3b21" exitCode=1 Jul 17 10:03:23.254271 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:23.253901 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerDied","Data":"952cc8767575ee677829afbc8e3e9941cf78c898eaff728b2b28fcf86e2c3b21"} Jul 17 10:03:23.254271 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:23.254130 2584 scope.go:117] "RemoveContainer" containerID="952cc8767575ee677829afbc8e3e9941cf78c898eaff728b2b28fcf86e2c3b21" Jul 17 10:03:24.265886 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:24.265785 2584 generic.go:356] "Generic (PLEG): container finished" podID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerID="12890e88b4eb48a1b122dfd7df985a75195ab5bce4b52b946c4dde2c820e706f" exitCode=1 Jul 17 10:03:24.266999 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:24.266971 2584 scope.go:117] "RemoveContainer" containerID="12890e88b4eb48a1b122dfd7df985a75195ab5bce4b52b946c4dde2c820e706f" Jul 17 10:03:24.267211 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:24.267182 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:03:24.267498 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:24.267473 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerDied","Data":"12890e88b4eb48a1b122dfd7df985a75195ab5bce4b52b946c4dde2c820e706f"} Jul 17 10:03:24.267578 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:24.267535 2584 scope.go:117] "RemoveContainer" containerID="952cc8767575ee677829afbc8e3e9941cf78c898eaff728b2b28fcf86e2c3b21" Jul 17 10:03:25.275914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:25.275889 2584 scope.go:117] "RemoveContainer" containerID="12890e88b4eb48a1b122dfd7df985a75195ab5bce4b52b946c4dde2c820e706f" Jul 17 10:03:25.276364 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:25.276132 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:03:26.806980 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:26.806938 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:26.807382 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:26.807370 2584 scope.go:117] "RemoveContainer" containerID="12890e88b4eb48a1b122dfd7df985a75195ab5bce4b52b946c4dde2c820e706f" Jul 17 10:03:26.807607 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:26.807584 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:03:31.807048 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:31.807010 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:31.807550 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:31.807452 2584 scope.go:117] "RemoveContainer" containerID="12890e88b4eb48a1b122dfd7df985a75195ab5bce4b52b946c4dde2c820e706f" Jul 17 10:03:31.807645 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:31.807625 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:03:32.313477 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:32.313440 2584 generic.go:356] "Generic (PLEG): container finished" podID="40223604-833e-43ef-85df-1873ee58eadc" containerID="fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f" exitCode=0 Jul 17 10:03:32.313477 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:32.313478 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" event={"ID":"40223604-833e-43ef-85df-1873ee58eadc","Type":"ContainerDied","Data":"fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f"} Jul 17 10:03:32.313851 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:32.313838 2584 scope.go:117] "RemoveContainer" containerID="fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f" Jul 17 10:03:32.314904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:32.314876 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:03:32.314957 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:32.314947 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:03:32.315003 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:32.314991 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:03:33.325914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:33.325875 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" event={"ID":"40223604-833e-43ef-85df-1873ee58eadc","Type":"ContainerStarted","Data":"f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af"} Jul 17 10:03:33.947038 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:33.946991 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:33.948056 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:33.948012 2584 patch_prober.go:28] interesting pod/e2e-derive-125842b9-openshift-default-677df58467-csvx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 17 10:03:33.948260 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:33.948095 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 17 10:03:34.946186 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:34.946137 2584 patch_prober.go:28] interesting pod/e2e-derive-125842b9-openshift-default-677df58467-csvx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 17 10:03:34.946689 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:34.946227 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 17 10:03:35.946689 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:35.946645 2584 patch_prober.go:28] interesting pod/e2e-derive-125842b9-openshift-default-677df58467-csvx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 17 10:03:35.948038 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:35.948000 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 17 10:03:36.946291 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:36.946249 2584 patch_prober.go:28] interesting pod/e2e-derive-125842b9-openshift-default-677df58467-csvx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 17 10:03:36.946647 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:36.946614 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 17 10:03:37.247695 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:37.247596 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:37.950259 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:37.950225 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:37.951314 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:37.951295 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:03:44.993917 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:44.993884 2584 scope.go:117] "RemoveContainer" containerID="12890e88b4eb48a1b122dfd7df985a75195ab5bce4b52b946c4dde2c820e706f" Jul 17 10:03:46.403245 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:46.403215 2584 generic.go:356] "Generic (PLEG): container finished" podID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerID="ae303a6230af1eceae2b2a167869bd9bae8e2f44e480fecc50d2bfa6783960b0" exitCode=1 Jul 17 10:03:46.403677 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:46.403290 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerDied","Data":"ae303a6230af1eceae2b2a167869bd9bae8e2f44e480fecc50d2bfa6783960b0"} Jul 17 10:03:46.403677 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:46.403358 2584 scope.go:117] "RemoveContainer" containerID="12890e88b4eb48a1b122dfd7df985a75195ab5bce4b52b946c4dde2c820e706f" Jul 17 10:03:46.403782 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:46.403739 2584 scope.go:117] "RemoveContainer" containerID="ae303a6230af1eceae2b2a167869bd9bae8e2f44e480fecc50d2bfa6783960b0" Jul 17 10:03:46.403947 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:46.403926 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:03:46.807178 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:46.807095 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:47.410252 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:47.410224 2584 scope.go:117] "RemoveContainer" containerID="ae303a6230af1eceae2b2a167869bd9bae8e2f44e480fecc50d2bfa6783960b0" Jul 17 10:03:47.410717 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:47.410428 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:03:51.806655 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:51.806613 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:03:51.807173 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:03:51.807152 2584 scope.go:117] "RemoveContainer" containerID="ae303a6230af1eceae2b2a167869bd9bae8e2f44e480fecc50d2bfa6783960b0" Jul 17 10:03:51.807429 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:03:51.807377 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:04:06.994188 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:06.994158 2584 scope.go:117] "RemoveContainer" containerID="ae303a6230af1eceae2b2a167869bd9bae8e2f44e480fecc50d2bfa6783960b0" Jul 17 10:04:07.522665 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:07.522634 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerStarted","Data":"7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1"} Jul 17 10:04:07.522847 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:07.522831 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:04:07.543498 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:07.543446 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podStartSLOduration=45.215958267 podStartE2EDuration="47.543431758s" totalImagesPullingTime="2.327473491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:03:20 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:04:07.54042669 +0000 UTC m=+2230.079625659" watchObservedRunningTime="2026-07-17 10:04:07.543431758 +0000 UTC m=+2230.082630725" Jul 17 10:04:08.529783 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:08.529748 2584 generic.go:356] "Generic (PLEG): container finished" podID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerID="7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1" exitCode=1 Jul 17 10:04:08.530245 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:08.529826 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerDied","Data":"7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1"} Jul 17 10:04:08.530245 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:08.529878 2584 scope.go:117] "RemoveContainer" containerID="ae303a6230af1eceae2b2a167869bd9bae8e2f44e480fecc50d2bfa6783960b0" Jul 17 10:04:08.530245 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:08.530145 2584 scope.go:117] "RemoveContainer" containerID="7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1" Jul 17 10:04:08.530429 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:04:08.530361 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:04:09.536789 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:09.536759 2584 scope.go:117] "RemoveContainer" containerID="7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1" Jul 17 10:04:09.537273 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:04:09.536990 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:04:11.806773 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:11.806733 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:04:11.807209 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:11.807194 2584 scope.go:117] "RemoveContainer" containerID="7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1" Jul 17 10:04:11.807437 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:04:11.807385 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:04:22.994107 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:22.994030 2584 scope.go:117] "RemoveContainer" containerID="7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1" Jul 17 10:04:22.994534 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:04:22.994209 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:04:36.993713 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:36.993679 2584 scope.go:117] "RemoveContainer" containerID="7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1" Jul 17 10:04:36.994142 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:04:36.993870 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:04:50.993930 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:50.993900 2584 scope.go:117] "RemoveContainer" containerID="7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1" Jul 17 10:04:51.755519 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:51.755430 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerStarted","Data":"0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d"} Jul 17 10:04:51.755676 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:51.755652 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:04:52.762661 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:52.762632 2584 generic.go:356] "Generic (PLEG): container finished" podID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" exitCode=1 Jul 17 10:04:52.763120 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:52.762684 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerDied","Data":"0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d"} Jul 17 10:04:52.763120 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:52.762731 2584 scope.go:117] "RemoveContainer" containerID="7ec8bf6ddc471a2963547169f23c4626f6f2d07c8f539191c09f031848348ef1" Jul 17 10:04:52.763120 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:52.762962 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:04:52.763249 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:04:52.763166 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:04:53.770032 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:04:53.770005 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:04:53.770456 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:04:53.770188 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:05:01.806590 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:05:01.806556 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:05:01.806967 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:05:01.806882 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:05:01.807064 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:05:01.807043 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:05:16.993877 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:05:16.993842 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:05:16.994348 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:05:16.994053 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:05:28.993445 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:05:28.993386 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:05:28.993873 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:05:28.993602 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:05:42.993792 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:05:42.993760 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:05:42.994190 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:05:42.993948 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:05:56.993424 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:05:56.993323 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:05:56.993862 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:05:56.993569 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:06:07.996105 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:07.996062 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:06:07.996647 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:06:07.996267 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:06:18.993474 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:18.993440 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:06:20.245343 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:20.245310 2584 generic.go:356] "Generic (PLEG): container finished" podID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerID="f401f057684bafd8cd125f4df81129ea5a7bc2856eb828338dfdf5262a854c0b" exitCode=1 Jul 17 10:06:20.245752 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:20.245384 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerDied","Data":"f401f057684bafd8cd125f4df81129ea5a7bc2856eb828338dfdf5262a854c0b"} Jul 17 10:06:20.245752 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:20.245467 2584 scope.go:117] "RemoveContainer" containerID="0389d3d630d151b1c8178850d8afa20ed9039381988768f822bd4ac78adae41d" Jul 17 10:06:20.245850 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:20.245781 2584 scope.go:117] "RemoveContainer" containerID="f401f057684bafd8cd125f4df81129ea5a7bc2856eb828338dfdf5262a854c0b" Jul 17 10:06:20.246034 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:06:20.246014 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:06:21.755916 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:21.755884 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:06:21.756292 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:21.756235 2584 scope.go:117] "RemoveContainer" containerID="f401f057684bafd8cd125f4df81129ea5a7bc2856eb828338dfdf5262a854c0b" Jul 17 10:06:21.756512 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:06:21.756490 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:06:21.806569 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:21.806540 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:06:22.257106 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:22.257082 2584 scope.go:117] "RemoveContainer" containerID="f401f057684bafd8cd125f4df81129ea5a7bc2856eb828338dfdf5262a854c0b" Jul 17 10:06:22.257294 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:06:22.257265 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb_odh-ai-gateway-infra(79e141fd-cd0a-4b37-a576-f318fee3308d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" Jul 17 10:06:26.253755 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:26.253719 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7"] Jul 17 10:06:26.254159 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:26.253934 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" containerID="cri-o://f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af" gracePeriod=30 Jul 17 10:06:31.507185 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.507159 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:06:31.607676 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.607648 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-certs\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.607812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.607708 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-socket\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.607812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.607739 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-envoy\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.607812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.607780 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40223604-833e-43ef-85df-1873ee58eadc-istio-podinfo\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.608029 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.607825 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-data\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.608029 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.607873 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8rvr\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-kube-api-access-d8rvr\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.608029 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.607919 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:06:31.608029 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.607978 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40223604-833e-43ef-85df-1873ee58eadc-istiod-ca-cert\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.608244 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.608046 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:06:31.608244 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.608054 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-istio-token\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.608244 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.608155 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-credential-socket\") pod \"40223604-833e-43ef-85df-1873ee58eadc\" (UID: \"40223604-833e-43ef-85df-1873ee58eadc\") " Jul 17 10:06:31.608526 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.608485 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:31.608526 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.608506 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:31.608700 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.608681 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:06:31.608764 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.608691 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40223604-833e-43ef-85df-1873ee58eadc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:06:31.608821 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.608801 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-data" (OuterVolumeSpecName: "istio-data") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:06:31.610099 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.610072 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-istio-token" (OuterVolumeSpecName: "istio-token") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:06:31.610388 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.610368 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:06:31.610513 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.610486 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-kube-api-access-d8rvr" (OuterVolumeSpecName: "kube-api-access-d8rvr") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "kube-api-access-d8rvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:06:31.610597 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.610574 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40223604-833e-43ef-85df-1873ee58eadc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40223604-833e-43ef-85df-1873ee58eadc" (UID: "40223604-833e-43ef-85df-1873ee58eadc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 10:06:31.709754 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.709666 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rvr\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-kube-api-access-d8rvr\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:31.709754 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.709693 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40223604-833e-43ef-85df-1873ee58eadc-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:31.709754 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.709702 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40223604-833e-43ef-85df-1873ee58eadc-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:31.709754 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.709712 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:31.709754 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.709721 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:31.709754 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.709728 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40223604-833e-43ef-85df-1873ee58eadc-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:31.709754 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:31.709736 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40223604-833e-43ef-85df-1873ee58eadc-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:32.032921 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.032885 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb"] Jul 17 10:06:32.155142 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.155119 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:06:32.213880 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.213843 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mhr2\" (UniqueName: \"kubernetes.io/projected/79e141fd-cd0a-4b37-a576-f318fee3308d-kube-api-access-6mhr2\") pod \"79e141fd-cd0a-4b37-a576-f318fee3308d\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " Jul 17 10:06:32.214067 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.213906 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e141fd-cd0a-4b37-a576-f318fee3308d-openshift-service-ca\") pod \"79e141fd-cd0a-4b37-a576-f318fee3308d\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " Jul 17 10:06:32.214067 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.213983 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls\") pod \"79e141fd-cd0a-4b37-a576-f318fee3308d\" (UID: \"79e141fd-cd0a-4b37-a576-f318fee3308d\") " Jul 17 10:06:32.214314 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.214285 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e141fd-cd0a-4b37-a576-f318fee3308d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79e141fd-cd0a-4b37-a576-f318fee3308d" (UID: "79e141fd-cd0a-4b37-a576-f318fee3308d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:06:32.216030 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.216003 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79e141fd-cd0a-4b37-a576-f318fee3308d" (UID: "79e141fd-cd0a-4b37-a576-f318fee3308d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:06:32.216123 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.216044 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e141fd-cd0a-4b37-a576-f318fee3308d-kube-api-access-6mhr2" (OuterVolumeSpecName: "kube-api-access-6mhr2") pod "79e141fd-cd0a-4b37-a576-f318fee3308d" (UID: "79e141fd-cd0a-4b37-a576-f318fee3308d"). InnerVolumeSpecName "kube-api-access-6mhr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:06:32.313259 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.313227 2584 generic.go:356] "Generic (PLEG): container finished" podID="40223604-833e-43ef-85df-1873ee58eadc" containerID="f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af" exitCode=0 Jul 17 10:06:32.313427 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.313280 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" Jul 17 10:06:32.313427 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.313305 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" event={"ID":"40223604-833e-43ef-85df-1873ee58eadc","Type":"ContainerDied","Data":"f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af"} Jul 17 10:06:32.313427 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.313368 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7" event={"ID":"40223604-833e-43ef-85df-1873ee58eadc","Type":"ContainerDied","Data":"fa6c709d2e22aad1d45db0f3fd2631b22573da20e02cb37498929973a29d15e5"} Jul 17 10:06:32.313427 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.313417 2584 scope.go:117] "RemoveContainer" containerID="f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af" Jul 17 10:06:32.314730 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.314710 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e141fd-cd0a-4b37-a576-f318fee3308d-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:32.314812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.314730 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" Jul 17 10:06:32.314812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.314736 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e141fd-cd0a-4b37-a576-f318fee3308d-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:32.314812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.314750 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mhr2\" (UniqueName: \"kubernetes.io/projected/79e141fd-cd0a-4b37-a576-f318fee3308d-kube-api-access-6mhr2\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:06:32.314812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.314761 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb" event={"ID":"79e141fd-cd0a-4b37-a576-f318fee3308d","Type":"ContainerDied","Data":"61f391fa3865a6aa7c3ff6cd56c27b7846944002259ddd79a7f3fa047e828b44"} Jul 17 10:06:32.325635 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.325617 2584 scope.go:117] "RemoveContainer" containerID="fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f" Jul 17 10:06:32.333474 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.333456 2584 scope.go:117] "RemoveContainer" containerID="f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af" Jul 17 10:06:32.333721 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:06:32.333704 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af\": container with ID starting with f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af not found: ID does not exist" containerID="f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af" Jul 17 10:06:32.333782 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.333727 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af"} err="failed to get container status \"f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af\": rpc error: code = NotFound desc = could not find container \"f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af\": container with ID starting with f154bbfb3d05916675926f8bbf067ea5fc6c811f70af23856383b0fafaaec2af not found: ID does not exist" Jul 17 10:06:32.333782 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.333741 2584 scope.go:117] "RemoveContainer" containerID="fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f" Jul 17 10:06:32.333970 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:06:32.333949 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f\": container with ID starting with fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f not found: ID does not exist" containerID="fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f" Jul 17 10:06:32.334011 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.333977 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f"} err="failed to get container status \"fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f\": rpc error: code = NotFound desc = could not find container \"fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f\": container with ID starting with fc3b3e1b18ec68aec50a97178569669597467e77cfbdddf75b03c57198ed346f not found: ID does not exist" Jul 17 10:06:32.334011 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.333994 2584 scope.go:117] "RemoveContainer" containerID="f401f057684bafd8cd125f4df81129ea5a7bc2856eb828338dfdf5262a854c0b" Jul 17 10:06:32.338406 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.338373 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7"] Jul 17 10:06:32.345540 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.345516 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-125842b9-openshift-default-677df58467-csvx7"] Jul 17 10:06:32.358532 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.358509 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb"] Jul 17 10:06:32.361105 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:32.361086 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-125842b9-94bcdd89f-wpzsb"] Jul 17 10:06:33.998958 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:33.998927 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40223604-833e-43ef-85df-1873ee58eadc" path="/var/lib/kubelet/pods/40223604-833e-43ef-85df-1873ee58eadc/volumes" Jul 17 10:06:33.999337 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:33.999322 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" path="/var/lib/kubelet/pods/79e141fd-cd0a-4b37-a576-f318fee3308d/volumes" Jul 17 10:06:58.140697 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:58.140666 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:06:58.143205 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:06:58.143186 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:08:17.043344 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043305 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf"] Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043718 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043730 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043745 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043750 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043757 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043763 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043769 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043774 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043779 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043784 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043802 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043807 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043859 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043867 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043873 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" Jul 17 10:08:17.043872 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043880 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.044381 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043886 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.044381 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043891 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.044381 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.043898 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:17.046915 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.046899 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.054145 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.054124 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cda038a0-openshift-default-dockercfg-ldrrj\"" Jul 17 10:08:17.079078 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.079047 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf"] Jul 17 10:08:17.209380 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209341 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-podinfo\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.209557 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209427 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-envoy\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.209557 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209453 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-data\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.209557 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209469 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istiod-ca-cert\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.209557 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209498 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-credential-socket\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.209696 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209577 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-token\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.209696 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209623 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-socket\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.209762 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209692 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-certs\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.209762 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.209747 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwhdw\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-kube-api-access-lwhdw\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310346 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-certs\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310392 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwhdw\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-kube-api-access-lwhdw\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310447 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-podinfo\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310718 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310488 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-envoy\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310718 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310605 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-data\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310718 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310641 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istiod-ca-cert\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310887 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310730 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-certs\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310887 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310846 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-credential-socket\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.310887 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310883 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-token\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.311035 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310931 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-socket\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.311035 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.310960 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-data\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.311195 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.311174 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-credential-socket\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.311276 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.311256 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-socket\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.311384 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.311369 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istiod-ca-cert\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.312844 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.312824 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-envoy\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.312989 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.312972 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-podinfo\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.319461 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.319436 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwhdw\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-kube-api-access-lwhdw\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.319586 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.319571 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-token\") pod \"e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.357917 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.357893 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:17.496512 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.496488 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf"] Jul 17 10:08:17.498591 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:08:17.498566 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58e01f9_feeb_4482_ad5e_dcc03c8dccf3.slice/crio-e8368f0d10eff0fa375fd41ca33b5469f045d50295a9ce06dfb6f2dd094b3b99 WatchSource:0}: Error finding container e8368f0d10eff0fa375fd41ca33b5469f045d50295a9ce06dfb6f2dd094b3b99: Status 404 returned error can't find the container with id e8368f0d10eff0fa375fd41ca33b5469f045d50295a9ce06dfb6f2dd094b3b99 Jul 17 10:08:17.501258 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.501227 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:08:17.501353 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.501289 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:08:17.501353 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.501319 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:08:17.868534 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.868497 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" event={"ID":"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3","Type":"ContainerStarted","Data":"b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d"} Jul 17 10:08:17.868743 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.868555 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" event={"ID":"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3","Type":"ContainerStarted","Data":"e8368f0d10eff0fa375fd41ca33b5469f045d50295a9ce06dfb6f2dd094b3b99"} Jul 17 10:08:17.907165 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:17.907094 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" podStartSLOduration=0.907070366 podStartE2EDuration="907.070366ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:08:17.904698506 +0000 UTC m=+2480.443897472" watchObservedRunningTime="2026-07-17 10:08:17.907070366 +0000 UTC m=+2480.446269332" Jul 17 10:08:18.358640 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:18.358608 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:18.363592 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:18.363568 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:18.874348 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:18.874318 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:18.875286 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:18.875269 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:20.357422 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.357377 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2"] Jul 17 10:08:20.357797 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.357748 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:20.357797 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.357759 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:20.357797 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.357773 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" Jul 17 10:08:20.357797 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.357780 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="40223604-833e-43ef-85df-1873ee58eadc" containerName="istio-proxy" Jul 17 10:08:20.357937 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.357852 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e141fd-cd0a-4b37-a576-f318fee3308d" containerName="maas-api" Jul 17 10:08:20.360887 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.360871 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.366221 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.366202 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cda038a0-serving-cert\"" Jul 17 10:08:20.382829 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.382805 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2"] Jul 17 10:08:20.543204 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.543164 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-maas-api-tls\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.543435 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.543240 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mldp\" (UniqueName: \"kubernetes.io/projected/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-kube-api-access-4mldp\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.543435 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.543306 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-openshift-service-ca\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.644167 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.644082 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-openshift-service-ca\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.644303 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.644180 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-maas-api-tls\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.644303 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.644216 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mldp\" (UniqueName: \"kubernetes.io/projected/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-kube-api-access-4mldp\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.644877 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.644856 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-openshift-service-ca\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.646573 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.646552 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-maas-api-tls\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.655807 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.655782 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mldp\" (UniqueName: \"kubernetes.io/projected/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-kube-api-access-4mldp\") pod \"maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.671492 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.671469 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:20.829894 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.829867 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2"] Jul 17 10:08:20.831600 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:08:20.831576 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcbf448_80a3_4e29_bcc0_b6f95cc15836.slice/crio-7a6c2a49835b7efa4c4a20f177ded0ce4a31df5ae5a503a4d9673ba45729f8a7 WatchSource:0}: Error finding container 7a6c2a49835b7efa4c4a20f177ded0ce4a31df5ae5a503a4d9673ba45729f8a7: Status 404 returned error can't find the container with id 7a6c2a49835b7efa4c4a20f177ded0ce4a31df5ae5a503a4d9673ba45729f8a7 Jul 17 10:08:20.833279 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.833259 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 10:08:20.885817 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:20.885768 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" event={"ID":"1dcbf448-80a3-4e29-bcc0-b6f95cc15836","Type":"ContainerStarted","Data":"7a6c2a49835b7efa4c4a20f177ded0ce4a31df5ae5a503a4d9673ba45729f8a7"} Jul 17 10:08:21.892849 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:21.892816 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" event={"ID":"1dcbf448-80a3-4e29-bcc0-b6f95cc15836","Type":"ContainerStarted","Data":"d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b"} Jul 17 10:08:21.893232 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:21.892943 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:21.916716 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:21.916669 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" podStartSLOduration=1.344914397 podStartE2EDuration="1.916654279s" totalImagesPullingTime="571.739882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:08:21.914605686 +0000 UTC m=+2484.453804653" watchObservedRunningTime="2026-07-17 10:08:21.916654279 +0000 UTC m=+2484.455853245" Jul 17 10:08:27.902965 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:27.902936 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:33.875607 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:33.875574 2584 patch_prober.go:28] interesting pod/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 17 10:08:33.875982 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:33.875639 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 17 10:08:33.965791 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:33.965708 2584 generic.go:356] "Generic (PLEG): container finished" podID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerID="b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d" exitCode=0 Jul 17 10:08:33.965943 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:33.965784 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" event={"ID":"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3","Type":"ContainerDied","Data":"b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d"} Jul 17 10:08:33.966203 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:33.966186 2584 scope.go:117] "RemoveContainer" containerID="b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d" Jul 17 10:08:33.967160 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:33.967134 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:08:33.967252 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:33.967214 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:08:33.967313 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:33.967297 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:08:34.973434 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:34.973375 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" event={"ID":"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3","Type":"ContainerStarted","Data":"43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6"} Jul 17 10:08:35.359200 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:35.359160 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:35.359443 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:35.359288 2584 patch_prober.go:28] interesting pod/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 17 10:08:35.359443 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:35.359330 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 17 10:08:36.358848 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:36.358807 2584 patch_prober.go:28] interesting pod/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 17 10:08:36.359486 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:36.359452 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 17 10:08:37.359048 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:37.359003 2584 patch_prober.go:28] interesting pod/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 17 10:08:37.359687 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:37.359654 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 17 10:08:38.359186 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:38.359149 2584 patch_prober.go:28] interesting pod/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 17 10:08:38.359834 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:38.359802 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 17 10:08:39.362866 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:39.362832 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:39.363322 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:39.363083 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:39.363846 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:39.363830 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:08:42.438472 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.438440 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2"] Jul 17 10:08:42.438914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.438650 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" podUID="1dcbf448-80a3-4e29-bcc0-b6f95cc15836" containerName="maas-api" containerID="cri-o://d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b" gracePeriod=30 Jul 17 10:08:42.687658 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.687632 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:42.767233 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.767140 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-maas-api-tls\") pod \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " Jul 17 10:08:42.767233 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.767183 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-openshift-service-ca\") pod \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " Jul 17 10:08:42.767233 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.767232 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mldp\" (UniqueName: \"kubernetes.io/projected/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-kube-api-access-4mldp\") pod \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\" (UID: \"1dcbf448-80a3-4e29-bcc0-b6f95cc15836\") " Jul 17 10:08:42.767671 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.767641 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1dcbf448-80a3-4e29-bcc0-b6f95cc15836" (UID: "1dcbf448-80a3-4e29-bcc0-b6f95cc15836"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:08:42.769382 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.769363 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1dcbf448-80a3-4e29-bcc0-b6f95cc15836" (UID: "1dcbf448-80a3-4e29-bcc0-b6f95cc15836"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:08:42.769498 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.769473 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-kube-api-access-4mldp" (OuterVolumeSpecName: "kube-api-access-4mldp") pod "1dcbf448-80a3-4e29-bcc0-b6f95cc15836" (UID: "1dcbf448-80a3-4e29-bcc0-b6f95cc15836"). InnerVolumeSpecName "kube-api-access-4mldp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:08:42.868596 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.868555 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mldp\" (UniqueName: \"kubernetes.io/projected/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-kube-api-access-4mldp\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:08:42.868596 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.868587 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:08:42.868596 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:42.868597 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf448-80a3-4e29-bcc0-b6f95cc15836-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:08:43.014969 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.014933 2584 generic.go:356] "Generic (PLEG): container finished" podID="1dcbf448-80a3-4e29-bcc0-b6f95cc15836" containerID="d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b" exitCode=0 Jul 17 10:08:43.015144 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.014982 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" Jul 17 10:08:43.015144 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.015015 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" event={"ID":"1dcbf448-80a3-4e29-bcc0-b6f95cc15836","Type":"ContainerDied","Data":"d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b"} Jul 17 10:08:43.015144 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.015061 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2" event={"ID":"1dcbf448-80a3-4e29-bcc0-b6f95cc15836","Type":"ContainerDied","Data":"7a6c2a49835b7efa4c4a20f177ded0ce4a31df5ae5a503a4d9673ba45729f8a7"} Jul 17 10:08:43.015144 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.015084 2584 scope.go:117] "RemoveContainer" containerID="d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b" Jul 17 10:08:43.024177 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.024162 2584 scope.go:117] "RemoveContainer" containerID="d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b" Jul 17 10:08:43.024463 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:08:43.024441 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b\": container with ID starting with d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b not found: ID does not exist" containerID="d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b" Jul 17 10:08:43.024523 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.024470 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b"} err="failed to get container status \"d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b\": rpc error: code = NotFound desc = could not find container \"d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b\": container with ID starting with d0f39149c9cad00f8fc6d7addf3dc849ebceca61035d6f9c130bde99956e513b not found: ID does not exist" Jul 17 10:08:43.041011 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.040990 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2"] Jul 17 10:08:43.049089 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.049055 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cda038a0-6df9cfbb84-thfh2"] Jul 17 10:08:43.998485 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:43.998453 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dcbf448-80a3-4e29-bcc0-b6f95cc15836" path="/var/lib/kubelet/pods/1dcbf448-80a3-4e29-bcc0-b6f95cc15836/volumes" Jul 17 10:08:57.766598 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:57.762229 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf"] Jul 17 10:08:57.766598 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:08:57.765309 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" containerID="cri-o://43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6" gracePeriod=30 Jul 17 10:09:03.015659 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.015637 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:09:03.127941 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.127905 2584 generic.go:356] "Generic (PLEG): container finished" podID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerID="43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6" exitCode=0 Jul 17 10:09:03.128158 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.127962 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" Jul 17 10:09:03.128158 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.127966 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" event={"ID":"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3","Type":"ContainerDied","Data":"43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6"} Jul 17 10:09:03.128158 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.128101 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf" event={"ID":"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3","Type":"ContainerDied","Data":"e8368f0d10eff0fa375fd41ca33b5469f045d50295a9ce06dfb6f2dd094b3b99"} Jul 17 10:09:03.128158 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.128123 2584 scope.go:117] "RemoveContainer" containerID="43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6" Jul 17 10:09:03.136847 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.136826 2584 scope.go:117] "RemoveContainer" containerID="b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d" Jul 17 10:09:03.144557 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.144540 2584 scope.go:117] "RemoveContainer" containerID="43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6" Jul 17 10:09:03.144791 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:09:03.144774 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6\": container with ID starting with 43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6 not found: ID does not exist" containerID="43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6" Jul 17 10:09:03.144842 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.144799 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6"} err="failed to get container status \"43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6\": rpc error: code = NotFound desc = could not find container \"43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6\": container with ID starting with 43057a77de38e8c170aa3cb892f98e2505d2cc7871999e27e80b09560b9c21f6 not found: ID does not exist" Jul 17 10:09:03.144842 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.144814 2584 scope.go:117] "RemoveContainer" containerID="b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d" Jul 17 10:09:03.145052 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:09:03.145035 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d\": container with ID starting with b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d not found: ID does not exist" containerID="b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d" Jul 17 10:09:03.145098 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.145058 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d"} err="failed to get container status \"b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d\": rpc error: code = NotFound desc = could not find container \"b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d\": container with ID starting with b0337ec2142f49e45029b4322bb90262a7eb42127e88be80e49c133f6656e72d not found: ID does not exist" Jul 17 10:09:03.157363 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157344 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-token\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157460 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157380 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-podinfo\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157460 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157419 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-socket\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157460 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157444 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-certs\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157609 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157472 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-envoy\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157609 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157497 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-data\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157609 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157542 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwhdw\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-kube-api-access-lwhdw\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157609 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157566 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istiod-ca-cert\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157806 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157611 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-credential-socket\") pod \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\" (UID: \"a58e01f9-feeb-4482-ad5e-dcc03c8dccf3\") " Jul 17 10:09:03.157806 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157668 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:09:03.157925 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.157803 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:09:03.158055 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.158011 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:09:03.158262 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.158240 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.158337 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.158271 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.158595 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.158567 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:09:03.158715 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.158642 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-data" (OuterVolumeSpecName: "istio-data") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:09:03.159626 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.159601 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-token" (OuterVolumeSpecName: "istio-token") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:09:03.159826 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.159808 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 10:09:03.160003 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.159976 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:09:03.160093 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.159990 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-kube-api-access-lwhdw" (OuterVolumeSpecName: "kube-api-access-lwhdw") pod "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" (UID: "a58e01f9-feeb-4482-ad5e-dcc03c8dccf3"). InnerVolumeSpecName "kube-api-access-lwhdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:09:03.259228 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.259192 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.259228 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.259221 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.259228 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.259231 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.259470 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.259239 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.259470 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.259248 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwhdw\" (UniqueName: \"kubernetes.io/projected/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-kube-api-access-lwhdw\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.259470 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.259258 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.259470 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.259268 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:09:03.454033 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.454004 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf"] Jul 17 10:09:03.458657 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.458634 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cda038a0-openshift-default-6f6958b475-hjbnf"] Jul 17 10:09:03.999175 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:03.999141 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" path="/var/lib/kubelet/pods/a58e01f9-feeb-4482-ad5e-dcc03c8dccf3/volumes" Jul 17 10:09:17.812617 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.812581 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b"] Jul 17 10:09:17.812982 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.812948 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" Jul 17 10:09:17.812982 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.812959 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" Jul 17 10:09:17.812982 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.812973 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dcbf448-80a3-4e29-bcc0-b6f95cc15836" containerName="maas-api" Jul 17 10:09:17.812982 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.812978 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dcbf448-80a3-4e29-bcc0-b6f95cc15836" containerName="maas-api" Jul 17 10:09:17.813109 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.812989 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" Jul 17 10:09:17.813109 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.812995 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" Jul 17 10:09:17.813109 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.813047 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dcbf448-80a3-4e29-bcc0-b6f95cc15836" containerName="maas-api" Jul 17 10:09:17.813109 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.813054 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" Jul 17 10:09:17.813109 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.813061 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a58e01f9-feeb-4482-ad5e-dcc03c8dccf3" containerName="istio-proxy" Jul 17 10:09:17.816190 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.816172 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.818860 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.818841 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6bff328f-openshift-default-dockercfg-kftwb\"" Jul 17 10:09:17.837307 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.837281 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b"] Jul 17 10:09:17.880586 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880553 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpnx8\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-kube-api-access-wpnx8\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.880586 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880590 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef803ba-8ca0-4887-a861-7601de5ae446-istiod-ca-cert\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.880822 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880616 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-podinfo\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.880822 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880723 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-socket\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.880822 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880754 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-envoy\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.880822 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880802 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-data\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.880982 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880833 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-token\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.880982 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880901 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-credential-socket\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.880982 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.880916 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-certs\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.981749 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.981715 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-socket\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.981749 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.981750 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-envoy\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.981997 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.981777 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-data\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.981997 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.981900 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-token\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.981997 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.981986 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-credential-socket\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982162 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982006 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-certs\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982162 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982035 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpnx8\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-kube-api-access-wpnx8\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982162 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982068 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef803ba-8ca0-4887-a861-7601de5ae446-istiod-ca-cert\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982162 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982106 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-podinfo\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982370 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982169 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-data\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982370 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982210 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-socket\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982370 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982349 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-credential-socket\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982566 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982523 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-certs\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.982858 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.982836 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef803ba-8ca0-4887-a861-7601de5ae446-istiod-ca-cert\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.984302 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.984280 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-envoy\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.984497 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.984481 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-podinfo\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.990271 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.990251 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-token\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:17.990732 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:17.990717 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpnx8\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-kube-api-access-wpnx8\") pod \"e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:18.126238 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:18.126202 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:18.265595 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:18.265565 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b"] Jul 17 10:09:18.267369 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:09:18.267344 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef803ba_8ca0_4887_a861_7601de5ae446.slice/crio-a10512b242c0ae8bb4d01e8e49c23be33b195ce53a00032f8ce2d996ff30d251 WatchSource:0}: Error finding container a10512b242c0ae8bb4d01e8e49c23be33b195ce53a00032f8ce2d996ff30d251: Status 404 returned error can't find the container with id a10512b242c0ae8bb4d01e8e49c23be33b195ce53a00032f8ce2d996ff30d251 Jul 17 10:09:18.269527 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:18.269496 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:09:18.269619 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:18.269562 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:09:18.269619 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:18.269588 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:09:19.214076 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:19.214034 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" event={"ID":"9ef803ba-8ca0-4887-a861-7601de5ae446","Type":"ContainerStarted","Data":"7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e"} Jul 17 10:09:19.214525 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:19.214095 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" event={"ID":"9ef803ba-8ca0-4887-a861-7601de5ae446","Type":"ContainerStarted","Data":"a10512b242c0ae8bb4d01e8e49c23be33b195ce53a00032f8ce2d996ff30d251"} Jul 17 10:09:19.236065 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:19.236013 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" podStartSLOduration=2.235993876 podStartE2EDuration="2.235993876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:09:19.233979488 +0000 UTC m=+2541.773178470" watchObservedRunningTime="2026-07-17 10:09:19.235993876 +0000 UTC m=+2541.775192842" Jul 17 10:09:20.126958 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:20.126912 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:20.132160 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:20.132132 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:20.218967 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:20.218935 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:20.219806 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:20.219787 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:21.331017 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.330985 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4"] Jul 17 10:09:21.334774 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.334752 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:21.337611 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.337593 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6bff328f-serving-cert\"" Jul 17 10:09:21.346993 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.346971 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4"] Jul 17 10:09:21.416200 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.416164 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgkk\" (UniqueName: \"kubernetes.io/projected/aeff8b31-74be-40e8-91fe-09cdf763ffd3-kube-api-access-9zgkk\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:21.416364 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.416259 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeff8b31-74be-40e8-91fe-09cdf763ffd3-openshift-service-ca\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:21.416364 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.416288 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:21.517730 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.517695 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgkk\" (UniqueName: \"kubernetes.io/projected/aeff8b31-74be-40e8-91fe-09cdf763ffd3-kube-api-access-9zgkk\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:21.517948 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.517804 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeff8b31-74be-40e8-91fe-09cdf763ffd3-openshift-service-ca\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:21.517948 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.517838 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:21.518084 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:09:21.517956 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-serving-cert: secret "maas-api-e2e-mt-6bff328f-serving-cert" not found Jul 17 10:09:21.518084 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:09:21.518034 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls podName:aeff8b31-74be-40e8-91fe-09cdf763ffd3 nodeName:}" failed. No retries permitted until 2026-07-17 10:09:22.018013394 +0000 UTC m=+2544.557212341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls") pod "maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" (UID: "aeff8b31-74be-40e8-91fe-09cdf763ffd3") : secret "maas-api-e2e-mt-6bff328f-serving-cert" not found Jul 17 10:09:21.518497 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.518478 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeff8b31-74be-40e8-91fe-09cdf763ffd3-openshift-service-ca\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:21.528938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:21.528917 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgkk\" (UniqueName: \"kubernetes.io/projected/aeff8b31-74be-40e8-91fe-09cdf763ffd3-kube-api-access-9zgkk\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:22.023801 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:22.023766 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:22.026303 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:22.026277 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls\") pod \"maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:22.245973 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:22.245942 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:22.377461 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:22.377432 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4"] Jul 17 10:09:22.378352 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:09:22.378328 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeff8b31_74be_40e8_91fe_09cdf763ffd3.slice/crio-7c978dc9add9edb21f23e2282be947a8e7f7f15e12b281bff91af0abc3c055fd WatchSource:0}: Error finding container 7c978dc9add9edb21f23e2282be947a8e7f7f15e12b281bff91af0abc3c055fd: Status 404 returned error can't find the container with id 7c978dc9add9edb21f23e2282be947a8e7f7f15e12b281bff91af0abc3c055fd Jul 17 10:09:23.236010 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:23.235976 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" event={"ID":"aeff8b31-74be-40e8-91fe-09cdf763ffd3","Type":"ContainerStarted","Data":"cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6"} Jul 17 10:09:23.236209 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:23.236026 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" event={"ID":"aeff8b31-74be-40e8-91fe-09cdf763ffd3","Type":"ContainerStarted","Data":"7c978dc9add9edb21f23e2282be947a8e7f7f15e12b281bff91af0abc3c055fd"} Jul 17 10:09:23.236209 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:23.236098 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:23.286854 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:23.286799 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" podStartSLOduration=1.855779338 podStartE2EDuration="2.286779363s" totalImagesPullingTime="431.000025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:09:23.28399589 +0000 UTC m=+2545.823194877" watchObservedRunningTime="2026-07-17 10:09:23.286779363 +0000 UTC m=+2545.825978327" Jul 17 10:09:29.247128 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:29.247098 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:09:36.262553 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:36.262515 2584 patch_prober.go:28] interesting pod/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 17 10:09:36.263846 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:36.262592 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 17 10:09:37.313562 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:37.313532 2584 generic.go:356] "Generic (PLEG): container finished" podID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerID="7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e" exitCode=0 Jul 17 10:09:37.313947 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:37.313597 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" event={"ID":"9ef803ba-8ca0-4887-a861-7601de5ae446","Type":"ContainerDied","Data":"7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e"} Jul 17 10:09:37.314008 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:37.313991 2584 scope.go:117] "RemoveContainer" containerID="7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e" Jul 17 10:09:37.315052 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:37.315026 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:09:37.315137 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:37.315091 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:09:37.315137 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:37.315121 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:09:38.320758 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:38.320722 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" event={"ID":"9ef803ba-8ca0-4887-a861-7601de5ae446","Type":"ContainerStarted","Data":"b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41"} Jul 17 10:09:39.126752 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:39.126715 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:39.126978 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:39.126959 2584 patch_prober.go:28] interesting pod/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 17 10:09:39.127034 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:39.127015 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 17 10:09:40.126952 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:40.126917 2584 patch_prober.go:28] interesting pod/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 17 10:09:40.127327 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:40.126975 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 17 10:09:41.126984 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:41.126950 2584 patch_prober.go:28] interesting pod/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 17 10:09:41.127379 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:41.127003 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 17 10:09:42.130615 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:42.130584 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:42.131010 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:42.130919 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:09:42.131680 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:09:42.131663 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:10:14.629997 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:14.629964 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4"] Jul 17 10:10:14.630599 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:14.630226 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" podUID="aeff8b31-74be-40e8-91fe-09cdf763ffd3" containerName="maas-api" containerID="cri-o://cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6" gracePeriod=30 Jul 17 10:10:14.885298 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:14.885234 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:10:15.037714 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.037681 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeff8b31-74be-40e8-91fe-09cdf763ffd3-openshift-service-ca\") pod \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " Jul 17 10:10:15.037714 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.037729 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zgkk\" (UniqueName: \"kubernetes.io/projected/aeff8b31-74be-40e8-91fe-09cdf763ffd3-kube-api-access-9zgkk\") pod \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " Jul 17 10:10:15.037933 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.037756 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls\") pod \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\" (UID: \"aeff8b31-74be-40e8-91fe-09cdf763ffd3\") " Jul 17 10:10:15.038158 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.038126 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeff8b31-74be-40e8-91fe-09cdf763ffd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aeff8b31-74be-40e8-91fe-09cdf763ffd3" (UID: "aeff8b31-74be-40e8-91fe-09cdf763ffd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:10:15.039921 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.039898 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeff8b31-74be-40e8-91fe-09cdf763ffd3-kube-api-access-9zgkk" (OuterVolumeSpecName: "kube-api-access-9zgkk") pod "aeff8b31-74be-40e8-91fe-09cdf763ffd3" (UID: "aeff8b31-74be-40e8-91fe-09cdf763ffd3"). InnerVolumeSpecName "kube-api-access-9zgkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:10:15.039921 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.039901 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aeff8b31-74be-40e8-91fe-09cdf763ffd3" (UID: "aeff8b31-74be-40e8-91fe-09cdf763ffd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:10:15.139318 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.139218 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeff8b31-74be-40e8-91fe-09cdf763ffd3-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:10:15.139318 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.139252 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgkk\" (UniqueName: \"kubernetes.io/projected/aeff8b31-74be-40e8-91fe-09cdf763ffd3-kube-api-access-9zgkk\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:10:15.139318 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.139266 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeff8b31-74be-40e8-91fe-09cdf763ffd3-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:10:15.517650 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.517567 2584 generic.go:356] "Generic (PLEG): container finished" podID="aeff8b31-74be-40e8-91fe-09cdf763ffd3" containerID="cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6" exitCode=0 Jul 17 10:10:15.517650 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.517612 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" Jul 17 10:10:15.517833 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.517652 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" event={"ID":"aeff8b31-74be-40e8-91fe-09cdf763ffd3","Type":"ContainerDied","Data":"cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6"} Jul 17 10:10:15.517833 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.517706 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4" event={"ID":"aeff8b31-74be-40e8-91fe-09cdf763ffd3","Type":"ContainerDied","Data":"7c978dc9add9edb21f23e2282be947a8e7f7f15e12b281bff91af0abc3c055fd"} Jul 17 10:10:15.517833 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.517735 2584 scope.go:117] "RemoveContainer" containerID="cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6" Jul 17 10:10:15.528131 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.528108 2584 scope.go:117] "RemoveContainer" containerID="cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6" Jul 17 10:10:15.528385 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:10:15.528365 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6\": container with ID starting with cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6 not found: ID does not exist" containerID="cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6" Jul 17 10:10:15.528468 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.528412 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6"} err="failed to get container status \"cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6\": rpc error: code = NotFound desc = could not find container \"cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6\": container with ID starting with cd48dfb0ed8f2199521b8adbb9fbe574c6fdfb3c9b39223a85dce50d854026a6 not found: ID does not exist" Jul 17 10:10:15.539952 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.539921 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4"] Jul 17 10:10:15.543673 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:15.543649 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6bff328f-6569cb6f55-jq7l4"] Jul 17 10:10:16.000195 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:10:16.000166 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeff8b31-74be-40e8-91fe-09cdf763ffd3" path="/var/lib/kubelet/pods/aeff8b31-74be-40e8-91fe-09cdf763ffd3/volumes" Jul 17 10:11:11.573356 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:11.573321 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b"] Jul 17 10:11:11.573865 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:11.573633 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" containerID="cri-o://b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41" gracePeriod=30 Jul 17 10:11:12.133025 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:12.132990 2584 patch_prober.go:28] interesting pod/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 10:11:12.133184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:12.133084 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 10:11:16.818104 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.818072 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:11:16.837090 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.837002 2584 generic.go:356] "Generic (PLEG): container finished" podID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerID="b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41" exitCode=0 Jul 17 10:11:16.837090 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.837047 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" Jul 17 10:11:16.837090 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.837078 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" event={"ID":"9ef803ba-8ca0-4887-a861-7601de5ae446","Type":"ContainerDied","Data":"b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41"} Jul 17 10:11:16.837375 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.837164 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b" event={"ID":"9ef803ba-8ca0-4887-a861-7601de5ae446","Type":"ContainerDied","Data":"a10512b242c0ae8bb4d01e8e49c23be33b195ce53a00032f8ce2d996ff30d251"} Jul 17 10:11:16.837375 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.837199 2584 scope.go:117] "RemoveContainer" containerID="b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41" Jul 17 10:11:16.847736 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.847697 2584 scope.go:117] "RemoveContainer" containerID="7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e" Jul 17 10:11:16.856610 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.856590 2584 scope.go:117] "RemoveContainer" containerID="b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41" Jul 17 10:11:16.856868 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:11:16.856840 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41\": container with ID starting with b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41 not found: ID does not exist" containerID="b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41" Jul 17 10:11:16.856914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.856879 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41"} err="failed to get container status \"b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41\": rpc error: code = NotFound desc = could not find container \"b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41\": container with ID starting with b4f422d66223eb67679cfcab36c8f2757fb3e0fa65ce30e2a4a5649d5b8dab41 not found: ID does not exist" Jul 17 10:11:16.856914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.856897 2584 scope.go:117] "RemoveContainer" containerID="7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e" Jul 17 10:11:16.857147 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:11:16.857131 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e\": container with ID starting with 7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e not found: ID does not exist" containerID="7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e" Jul 17 10:11:16.857211 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.857149 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e"} err="failed to get container status \"7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e\": rpc error: code = NotFound desc = could not find container \"7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e\": container with ID starting with 7f33a3b386793946cc3e2aad77efe4837f45e3708400352f8bbf8660a4a7080e not found: ID does not exist" Jul 17 10:11:16.988340 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988310 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-certs\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988340 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988348 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-data\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988602 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988372 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-podinfo\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988602 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988463 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-credential-socket\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988602 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988508 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-socket\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988602 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988559 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef803ba-8ca0-4887-a861-7601de5ae446-istiod-ca-cert\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988602 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988601 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-envoy\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988859 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988646 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-token\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988859 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988680 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:11:16.988859 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988796 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpnx8\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-kube-api-access-wpnx8\") pod \"9ef803ba-8ca0-4887-a861-7601de5ae446\" (UID: \"9ef803ba-8ca0-4887-a861-7601de5ae446\") " Jul 17 10:11:16.988859 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988807 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:11:16.988859 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.988830 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:11:16.989130 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.989072 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef803ba-8ca0-4887-a861-7601de5ae446-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:11:16.989182 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.989150 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:16.989182 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.989170 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:16.989263 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.989199 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:16.989263 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.989215 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ef803ba-8ca0-4887-a861-7601de5ae446-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:16.989436 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.989412 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:11:16.990771 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.990743 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 10:11:16.990862 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.990834 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-kube-api-access-wpnx8" (OuterVolumeSpecName: "kube-api-access-wpnx8") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "kube-api-access-wpnx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:11:16.991171 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.991151 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:11:16.991171 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:16.991155 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ef803ba-8ca0-4887-a861-7601de5ae446" (UID: "9ef803ba-8ca0-4887-a861-7601de5ae446"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:11:17.090299 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:17.090275 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:17.090299 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:17.090301 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:17.090500 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:17.090310 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpnx8\" (UniqueName: \"kubernetes.io/projected/9ef803ba-8ca0-4887-a861-7601de5ae446-kube-api-access-wpnx8\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:17.090500 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:17.090319 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:17.090500 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:17.090328 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ef803ba-8ca0-4887-a861-7601de5ae446-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:11:17.162359 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:17.162331 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b"] Jul 17 10:11:17.168328 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:17.168303 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6bff328f-openshift-default-6bc6db696c-5hv6b"] Jul 17 10:11:17.999256 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:17.999223 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" path="/var/lib/kubelet/pods/9ef803ba-8ca0-4887-a861-7601de5ae446/volumes" Jul 17 10:11:58.176202 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:58.176137 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:11:58.180911 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:11:58.180895 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:12:06.966111 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.966067 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6"] Jul 17 10:12:06.966904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.966628 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" Jul 17 10:12:06.966904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.966646 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" Jul 17 10:12:06.966904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.966683 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeff8b31-74be-40e8-91fe-09cdf763ffd3" containerName="maas-api" Jul 17 10:12:06.966904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.966692 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeff8b31-74be-40e8-91fe-09cdf763ffd3" containerName="maas-api" Jul 17 10:12:06.966904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.966782 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeff8b31-74be-40e8-91fe-09cdf763ffd3" containerName="maas-api" Jul 17 10:12:06.966904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.966797 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" Jul 17 10:12:06.970091 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.970077 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:06.972838 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.972815 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-21e67a-openshift-default-dockercfg-4g4rv\"" Jul 17 10:12:06.981167 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:06.981148 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6"] Jul 17 10:12:07.146815 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.146780 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-certs\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.146815 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.146815 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72a4fd5-288d-43f1-bd78-bdfe86734322-istiod-ca-cert\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.147012 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.146900 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-credential-socket\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.147012 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.146941 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-podinfo\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.147012 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.146967 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wl5p\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-kube-api-access-8wl5p\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.147119 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.147018 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-socket\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.147119 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.147077 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-token\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.147194 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.147154 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-data\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.147238 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.147223 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-envoy\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.247950 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.247863 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-data\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.247950 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.247905 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-envoy\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248181 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.247948 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-certs\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248181 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.247973 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72a4fd5-288d-43f1-bd78-bdfe86734322-istiod-ca-cert\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248181 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248019 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-credential-socket\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248181 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248061 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-podinfo\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248181 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248084 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wl5p\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-kube-api-access-8wl5p\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248181 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248148 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-socket\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248530 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248214 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-token\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248530 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248294 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-data\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248530 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248392 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-certs\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248702 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248624 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-socket\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248752 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248710 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72a4fd5-288d-43f1-bd78-bdfe86734322-istiod-ca-cert\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.248752 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.248714 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-credential-socket\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.250232 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.250212 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-envoy\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.250674 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.250653 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-podinfo\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.256595 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.256572 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-token\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.256902 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.256885 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wl5p\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-kube-api-access-8wl5p\") pod \"e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.280301 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.280279 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:07.419566 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.419542 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6"] Jul 17 10:12:07.421250 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:12:07.421224 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72a4fd5_288d_43f1_bd78_bdfe86734322.slice/crio-9c569701bce77681d76c3f3f319bd8614844370f76b99e880521aa476a6e9242 WatchSource:0}: Error finding container 9c569701bce77681d76c3f3f319bd8614844370f76b99e880521aa476a6e9242: Status 404 returned error can't find the container with id 9c569701bce77681d76c3f3f319bd8614844370f76b99e880521aa476a6e9242 Jul 17 10:12:07.423360 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.423329 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:12:07.423457 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.423393 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:12:07.423504 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:07.423456 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:12:08.109276 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:08.109233 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" event={"ID":"c72a4fd5-288d-43f1-bd78-bdfe86734322","Type":"ContainerStarted","Data":"a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca"} Jul 17 10:12:08.109731 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:08.109292 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" event={"ID":"c72a4fd5-288d-43f1-bd78-bdfe86734322","Type":"ContainerStarted","Data":"9c569701bce77681d76c3f3f319bd8614844370f76b99e880521aa476a6e9242"} Jul 17 10:12:08.134428 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:08.133495 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" podStartSLOduration=2.133476221 podStartE2EDuration="2.133476221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:12:08.129597071 +0000 UTC m=+2710.668796038" watchObservedRunningTime="2026-07-17 10:12:08.133476221 +0000 UTC m=+2710.672675188" Jul 17 10:12:08.280832 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:08.280804 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:08.285272 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:08.285252 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:09.114051 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:09.114017 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:09.114928 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:09.114910 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:10.265240 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.265207 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj"] Jul 17 10:12:10.265669 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.265624 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" Jul 17 10:12:10.265669 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.265636 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" Jul 17 10:12:10.265743 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.265716 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef803ba-8ca0-4887-a861-7601de5ae446" containerName="istio-proxy" Jul 17 10:12:10.268753 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.268737 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.271975 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.271958 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-21e67a-serving-cert\"" Jul 17 10:12:10.273362 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.273345 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.273456 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.273382 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmvl2\" (UniqueName: \"kubernetes.io/projected/b1c72c57-f629-483c-8d11-32444a69a4d0-kube-api-access-kmvl2\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.273456 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.273452 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c72c57-f629-483c-8d11-32444a69a4d0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.279734 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.279712 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj"] Jul 17 10:12:10.374693 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.374655 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmvl2\" (UniqueName: \"kubernetes.io/projected/b1c72c57-f629-483c-8d11-32444a69a4d0-kube-api-access-kmvl2\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.374888 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.374706 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c72c57-f629-483c-8d11-32444a69a4d0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.374888 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.374812 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.375010 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:12:10.374917 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-serving-cert: secret "maas-api-e2e-inf-a-21e67a-serving-cert" not found Jul 17 10:12:10.375010 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:12:10.374968 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls podName:b1c72c57-f629-483c-8d11-32444a69a4d0 nodeName:}" failed. No retries permitted until 2026-07-17 10:12:10.874953717 +0000 UTC m=+2713.414152661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls") pod "maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" (UID: "b1c72c57-f629-483c-8d11-32444a69a4d0") : secret "maas-api-e2e-inf-a-21e67a-serving-cert" not found Jul 17 10:12:10.375293 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.375276 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c72c57-f629-483c-8d11-32444a69a4d0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.386284 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.386254 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmvl2\" (UniqueName: \"kubernetes.io/projected/b1c72c57-f629-483c-8d11-32444a69a4d0-kube-api-access-kmvl2\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.880172 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.880135 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:10.882626 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:10.882605 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls\") pod \"maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:11.180736 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:11.180659 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:11.325165 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:11.321765 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj"] Jul 17 10:12:12.129270 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:12.129172 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" event={"ID":"b1c72c57-f629-483c-8d11-32444a69a4d0","Type":"ContainerStarted","Data":"5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf"} Jul 17 10:12:12.129270 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:12.129239 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" event={"ID":"b1c72c57-f629-483c-8d11-32444a69a4d0","Type":"ContainerStarted","Data":"576d6d7436ad58f4361e4003c26d241c85ae610215843d99c8d50abf0eaaf66a"} Jul 17 10:12:12.129270 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:12.129269 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:12.145965 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:12.145923 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" podStartSLOduration=1.605473877 podStartE2EDuration="2.145908737s" totalImagesPullingTime="540.43486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:12:12.144790921 +0000 UTC m=+2714.683989893" watchObservedRunningTime="2026-07-17 10:12:12.145908737 +0000 UTC m=+2714.685107703" Jul 17 10:12:18.139800 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.139768 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:12:18.492728 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.492649 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df"] Jul 17 10:12:18.496495 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.496477 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.500535 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.500510 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1c996b-serving-cert\"" Jul 17 10:12:18.512622 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.512590 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df"] Jul 17 10:12:18.553257 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.553223 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.553450 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.553278 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.553500 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.553445 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sw8h\" (UniqueName: \"kubernetes.io/projected/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-kube-api-access-2sw8h\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.654465 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.654421 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sw8h\" (UniqueName: \"kubernetes.io/projected/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-kube-api-access-2sw8h\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.654648 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.654491 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.654648 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.654524 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.655177 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.655159 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.656929 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.656904 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-maas-api-tls\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.669924 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.669896 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sw8h\" (UniqueName: \"kubernetes.io/projected/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-kube-api-access-2sw8h\") pod \"maas-api-e2e-inf-b-1c996b-6755674f-6g2df\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.807372 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.807287 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:18.973867 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:18.973837 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df"] Jul 17 10:12:18.974201 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:12:18.974178 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9324f2e4_7c64_48ab_a4fc_5be9fb1ee595.slice/crio-6e6468717150da4d83994a869f77963e3d01b0461ac02cc7ba212a6bb8093c6d WatchSource:0}: Error finding container 6e6468717150da4d83994a869f77963e3d01b0461ac02cc7ba212a6bb8093c6d: Status 404 returned error can't find the container with id 6e6468717150da4d83994a869f77963e3d01b0461ac02cc7ba212a6bb8093c6d Jul 17 10:12:19.166080 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:19.166044 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" event={"ID":"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595","Type":"ContainerStarted","Data":"6e6468717150da4d83994a869f77963e3d01b0461ac02cc7ba212a6bb8093c6d"} Jul 17 10:12:20.173814 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:20.173766 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" event={"ID":"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595","Type":"ContainerStarted","Data":"d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11"} Jul 17 10:12:20.174204 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:20.173857 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:20.193467 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:20.193411 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" podStartSLOduration=1.908964423 podStartE2EDuration="2.193381893s" totalImagesPullingTime="284.41747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:12:20.190923979 +0000 UTC m=+2722.730122979" watchObservedRunningTime="2026-07-17 10:12:20.193381893 +0000 UTC m=+2722.732580858" Jul 17 10:12:26.186246 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:26.186219 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:12:27.213152 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:27.213069 2584 generic.go:356] "Generic (PLEG): container finished" podID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerID="a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca" exitCode=0 Jul 17 10:12:27.213152 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:27.213138 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" event={"ID":"c72a4fd5-288d-43f1-bd78-bdfe86734322","Type":"ContainerDied","Data":"a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca"} Jul 17 10:12:27.213633 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:27.213617 2584 scope.go:117] "RemoveContainer" containerID="a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca" Jul 17 10:12:27.214737 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:27.214703 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:12:27.214841 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:27.214773 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:12:27.214841 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:27.214811 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:12:28.221499 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:28.221462 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" event={"ID":"c72a4fd5-288d-43f1-bd78-bdfe86734322","Type":"ContainerStarted","Data":"53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8"} Jul 17 10:12:28.280677 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:28.280635 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:28.280991 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:28.280968 2584 patch_prober.go:28] interesting pod/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 17 10:12:28.281085 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:28.281042 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 17 10:12:29.281266 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:29.281225 2584 patch_prober.go:28] interesting pod/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 17 10:12:29.281710 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:29.281287 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 17 10:12:30.281484 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:30.281448 2584 patch_prober.go:28] interesting pod/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 17 10:12:30.281889 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:30.281505 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 17 10:12:31.281008 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:31.280971 2584 patch_prober.go:28] interesting pod/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 17 10:12:31.281179 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:31.281030 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 17 10:12:32.284675 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:32.284647 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:32.285055 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:32.285024 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:12:32.285617 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:12:32.285602 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:14:04.670760 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:04.670727 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj"] Jul 17 10:14:04.671252 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:04.670928 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" podUID="b1c72c57-f629-483c-8d11-32444a69a4d0" containerName="maas-api" containerID="cri-o://5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf" gracePeriod=30 Jul 17 10:14:04.920671 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:04.920642 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:14:05.078543 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.078440 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c72c57-f629-483c-8d11-32444a69a4d0-openshift-service-ca\") pod \"b1c72c57-f629-483c-8d11-32444a69a4d0\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " Jul 17 10:14:05.078543 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.078508 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls\") pod \"b1c72c57-f629-483c-8d11-32444a69a4d0\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " Jul 17 10:14:05.078543 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.078533 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmvl2\" (UniqueName: \"kubernetes.io/projected/b1c72c57-f629-483c-8d11-32444a69a4d0-kube-api-access-kmvl2\") pod \"b1c72c57-f629-483c-8d11-32444a69a4d0\" (UID: \"b1c72c57-f629-483c-8d11-32444a69a4d0\") " Jul 17 10:14:05.078856 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.078833 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c72c57-f629-483c-8d11-32444a69a4d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1c72c57-f629-483c-8d11-32444a69a4d0" (UID: "b1c72c57-f629-483c-8d11-32444a69a4d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:14:05.080689 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.080659 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1c72c57-f629-483c-8d11-32444a69a4d0" (UID: "b1c72c57-f629-483c-8d11-32444a69a4d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:14:05.080798 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.080782 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c72c57-f629-483c-8d11-32444a69a4d0-kube-api-access-kmvl2" (OuterVolumeSpecName: "kube-api-access-kmvl2") pod "b1c72c57-f629-483c-8d11-32444a69a4d0" (UID: "b1c72c57-f629-483c-8d11-32444a69a4d0"). InnerVolumeSpecName "kube-api-access-kmvl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:14:05.180281 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.180247 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1c72c57-f629-483c-8d11-32444a69a4d0-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:05.180281 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.180273 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1c72c57-f629-483c-8d11-32444a69a4d0-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:05.180281 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.180284 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmvl2\" (UniqueName: \"kubernetes.io/projected/b1c72c57-f629-483c-8d11-32444a69a4d0-kube-api-access-kmvl2\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:05.755107 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.755069 2584 generic.go:356] "Generic (PLEG): container finished" podID="b1c72c57-f629-483c-8d11-32444a69a4d0" containerID="5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf" exitCode=0 Jul 17 10:14:05.755107 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.755110 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" Jul 17 10:14:05.755693 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.755144 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" event={"ID":"b1c72c57-f629-483c-8d11-32444a69a4d0","Type":"ContainerDied","Data":"5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf"} Jul 17 10:14:05.755693 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.755203 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj" event={"ID":"b1c72c57-f629-483c-8d11-32444a69a4d0","Type":"ContainerDied","Data":"576d6d7436ad58f4361e4003c26d241c85ae610215843d99c8d50abf0eaaf66a"} Jul 17 10:14:05.755693 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.755226 2584 scope.go:117] "RemoveContainer" containerID="5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf" Jul 17 10:14:05.764651 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.764632 2584 scope.go:117] "RemoveContainer" containerID="5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf" Jul 17 10:14:05.764903 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:14:05.764884 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf\": container with ID starting with 5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf not found: ID does not exist" containerID="5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf" Jul 17 10:14:05.764955 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.764912 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf"} err="failed to get container status \"5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf\": rpc error: code = NotFound desc = could not find container \"5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf\": container with ID starting with 5b01262b1eed0d8c5940212c3167e39fa14e2f06011433c5efe03a221400d9bf not found: ID does not exist" Jul 17 10:14:05.776965 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.776943 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj"] Jul 17 10:14:05.781016 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.780995 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21e67a-897b476c8-wn2jj"] Jul 17 10:14:05.998952 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:05.998918 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1c72c57-f629-483c-8d11-32444a69a4d0" path="/var/lib/kubelet/pods/b1c72c57-f629-483c-8d11-32444a69a4d0/volumes" Jul 17 10:14:20.335744 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:20.335707 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6"] Jul 17 10:14:20.336141 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:20.335983 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" containerID="cri-o://53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8" gracePeriod=30 Jul 17 10:14:25.589870 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.589846 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:14:25.674970 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.674879 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-podinfo\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.674970 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.674933 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-token\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.674970 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.674962 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-certs\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.675260 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675002 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72a4fd5-288d-43f1-bd78-bdfe86734322-istiod-ca-cert\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.675260 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675038 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-data\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.675260 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675061 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-envoy\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.675260 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675078 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-credential-socket\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.675260 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675111 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wl5p\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-kube-api-access-8wl5p\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.675260 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675164 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-socket\") pod \"c72a4fd5-288d-43f1-bd78-bdfe86734322\" (UID: \"c72a4fd5-288d-43f1-bd78-bdfe86734322\") " Jul 17 10:14:25.675590 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675356 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:14:25.675590 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675507 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:14:25.675590 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675528 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72a4fd5-288d-43f1-bd78-bdfe86734322-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:14:25.675590 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675558 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.675776 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675626 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:14:25.675985 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.675960 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-data" (OuterVolumeSpecName: "istio-data") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:14:25.677353 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.677316 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 10:14:25.677353 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.677328 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-token" (OuterVolumeSpecName: "istio-token") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:14:25.677556 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.677408 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:14:25.677556 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.677424 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-kube-api-access-8wl5p" (OuterVolumeSpecName: "kube-api-access-8wl5p") pod "c72a4fd5-288d-43f1-bd78-bdfe86734322" (UID: "c72a4fd5-288d-43f1-bd78-bdfe86734322"). InnerVolumeSpecName "kube-api-access-8wl5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:14:25.776103 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.776070 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.776103 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.776100 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wl5p\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-kube-api-access-8wl5p\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.776103 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.776111 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.776350 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.776124 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.776350 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.776136 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.776350 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.776145 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c72a4fd5-288d-43f1-bd78-bdfe86734322-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.776350 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.776152 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.776350 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.776159 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c72a4fd5-288d-43f1-bd78-bdfe86734322-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:25.865679 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.865649 2584 generic.go:356] "Generic (PLEG): container finished" podID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerID="53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8" exitCode=0 Jul 17 10:14:25.865882 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.865705 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" Jul 17 10:14:25.865882 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.865741 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" event={"ID":"c72a4fd5-288d-43f1-bd78-bdfe86734322","Type":"ContainerDied","Data":"53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8"} Jul 17 10:14:25.865882 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.865809 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6" event={"ID":"c72a4fd5-288d-43f1-bd78-bdfe86734322","Type":"ContainerDied","Data":"9c569701bce77681d76c3f3f319bd8614844370f76b99e880521aa476a6e9242"} Jul 17 10:14:25.865882 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.865839 2584 scope.go:117] "RemoveContainer" containerID="53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8" Jul 17 10:14:25.875076 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.875060 2584 scope.go:117] "RemoveContainer" containerID="a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca" Jul 17 10:14:25.883730 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.883713 2584 scope.go:117] "RemoveContainer" containerID="53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8" Jul 17 10:14:25.883958 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:14:25.883936 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8\": container with ID starting with 53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8 not found: ID does not exist" containerID="53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8" Jul 17 10:14:25.884015 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.883966 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8"} err="failed to get container status \"53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8\": rpc error: code = NotFound desc = could not find container \"53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8\": container with ID starting with 53fabaf773c33f782ae156b38778d2450a7ef004f695529d74ed52951bba43f8 not found: ID does not exist" Jul 17 10:14:25.884015 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.883981 2584 scope.go:117] "RemoveContainer" containerID="a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca" Jul 17 10:14:25.884192 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:14:25.884177 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca\": container with ID starting with a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca not found: ID does not exist" containerID="a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca" Jul 17 10:14:25.884234 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.884197 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca"} err="failed to get container status \"a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca\": rpc error: code = NotFound desc = could not find container \"a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca\": container with ID starting with a7b8a1612293c9d8368dbc9c64ed601dd50e8abab2f550cbcf44b550b97c7eca not found: ID does not exist" Jul 17 10:14:25.887727 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.887706 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6"] Jul 17 10:14:25.891638 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.891613 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-21e67a-openshift-default-b4d8968f-r64q6"] Jul 17 10:14:25.999130 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:25.999060 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" path="/var/lib/kubelet/pods/c72a4fd5-288d-43f1-bd78-bdfe86734322/volumes" Jul 17 10:14:31.303659 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.303626 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df"] Jul 17 10:14:31.304137 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.303898 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" podUID="9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" containerName="maas-api" containerID="cri-o://d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11" gracePeriod=30 Jul 17 10:14:31.545756 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.545735 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:14:31.628287 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.628258 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-maas-api-tls\") pod \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " Jul 17 10:14:31.628443 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.628311 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sw8h\" (UniqueName: \"kubernetes.io/projected/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-kube-api-access-2sw8h\") pod \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " Jul 17 10:14:31.628443 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.628423 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-openshift-service-ca\") pod \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\" (UID: \"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595\") " Jul 17 10:14:31.628729 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.628707 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" (UID: "9324f2e4-7c64-48ab-a4fc-5be9fb1ee595"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:14:31.630376 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.630356 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-kube-api-access-2sw8h" (OuterVolumeSpecName: "kube-api-access-2sw8h") pod "9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" (UID: "9324f2e4-7c64-48ab-a4fc-5be9fb1ee595"). InnerVolumeSpecName "kube-api-access-2sw8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:14:31.630486 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.630457 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" (UID: "9324f2e4-7c64-48ab-a4fc-5be9fb1ee595"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:14:31.729324 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.729282 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:31.729324 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.729316 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:31.729324 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.729328 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sw8h\" (UniqueName: \"kubernetes.io/projected/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595-kube-api-access-2sw8h\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:14:31.898089 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.897994 2584 generic.go:356] "Generic (PLEG): container finished" podID="9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" containerID="d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11" exitCode=0 Jul 17 10:14:31.898089 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.898039 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" Jul 17 10:14:31.898342 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.898084 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" event={"ID":"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595","Type":"ContainerDied","Data":"d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11"} Jul 17 10:14:31.898342 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.898146 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df" event={"ID":"9324f2e4-7c64-48ab-a4fc-5be9fb1ee595","Type":"ContainerDied","Data":"6e6468717150da4d83994a869f77963e3d01b0461ac02cc7ba212a6bb8093c6d"} Jul 17 10:14:31.898342 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.898175 2584 scope.go:117] "RemoveContainer" containerID="d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11" Jul 17 10:14:31.907232 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.907216 2584 scope.go:117] "RemoveContainer" containerID="d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11" Jul 17 10:14:31.907519 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:14:31.907498 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11\": container with ID starting with d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11 not found: ID does not exist" containerID="d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11" Jul 17 10:14:31.907570 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.907528 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11"} err="failed to get container status \"d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11\": rpc error: code = NotFound desc = could not find container \"d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11\": container with ID starting with d2354f97b81c6edb30e3309c5d60574538876218cdef11e15b5936e4f90b8f11 not found: ID does not exist" Jul 17 10:14:31.920173 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.920145 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df"] Jul 17 10:14:31.924210 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.924192 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1c996b-6755674f-6g2df"] Jul 17 10:14:31.998720 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:31.998687 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" path="/var/lib/kubelet/pods/9324f2e4-7c64-48ab-a4fc-5be9fb1ee595/volumes" Jul 17 10:14:47.550723 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.550691 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67"] Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551042 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1c72c57-f629-483c-8d11-32444a69a4d0" containerName="maas-api" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551053 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1c72c57-f629-483c-8d11-32444a69a4d0" containerName="maas-api" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551063 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551069 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551083 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" containerName="maas-api" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551088 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" containerName="maas-api" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551099 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551104 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551170 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551177 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1c72c57-f629-483c-8d11-32444a69a4d0" containerName="maas-api" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551182 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c72a4fd5-288d-43f1-bd78-bdfe86734322" containerName="istio-proxy" Jul 17 10:14:47.551184 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.551188 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="9324f2e4-7c64-48ab-a4fc-5be9fb1ee595" containerName="maas-api" Jul 17 10:14:47.554155 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.554140 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.559960 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.559933 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d4c7ee-openshift-default-dockercfg-zp87j\"" Jul 17 10:14:47.566632 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.566563 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67"] Jul 17 10:14:47.670541 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670505 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-token\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.670732 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670556 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-podinfo\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.670732 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670624 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istiod-ca-cert\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.670732 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670656 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ldfm\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-kube-api-access-7ldfm\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.670732 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670713 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-credential-socket\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.670961 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670762 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-socket\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.670961 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670811 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-certs\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.670961 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670854 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-data\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.670961 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.670888 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-envoy\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772127 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772091 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-socket\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772304 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-certs\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772304 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772157 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-data\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772451 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772358 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-envoy\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772451 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772435 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-token\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772575 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772483 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-podinfo\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772575 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772496 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-data\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772575 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772560 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-certs\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772731 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772565 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istiod-ca-cert\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772731 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772591 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-socket\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772731 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772631 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ldfm\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-kube-api-access-7ldfm\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772731 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772670 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-credential-socket\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.772959 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.772943 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-credential-socket\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.773178 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.773156 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istiod-ca-cert\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.774590 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.774569 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-envoy\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.774981 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.774965 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-podinfo\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.783761 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.783740 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-token\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.787616 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.787594 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ldfm\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-kube-api-access-7ldfm\") pod \"e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:47.870877 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:47.870846 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:48.012013 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:48.011989 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67"] Jul 17 10:14:48.013279 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:14:48.013254 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a100ee_8398_4d0e_b77d_7dd2a090eb60.slice/crio-47abcb88a5802eb4defce449ee4822292cc26e37b4c5cdd51fa9e91017eec7d7 WatchSource:0}: Error finding container 47abcb88a5802eb4defce449ee4822292cc26e37b4c5cdd51fa9e91017eec7d7: Status 404 returned error can't find the container with id 47abcb88a5802eb4defce449ee4822292cc26e37b4c5cdd51fa9e91017eec7d7 Jul 17 10:14:48.015227 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:48.015201 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:14:48.015293 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:48.015260 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:14:48.015293 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:48.015285 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:14:48.994777 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:48.994742 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" event={"ID":"f8a100ee-8398-4d0e-b77d-7dd2a090eb60","Type":"ContainerStarted","Data":"a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba"} Jul 17 10:14:48.995163 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:48.994803 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" event={"ID":"f8a100ee-8398-4d0e-b77d-7dd2a090eb60","Type":"ContainerStarted","Data":"47abcb88a5802eb4defce449ee4822292cc26e37b4c5cdd51fa9e91017eec7d7"} Jul 17 10:14:49.017987 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:49.017943 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" podStartSLOduration=2.017929908 podStartE2EDuration="2.017929908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:14:49.01520494 +0000 UTC m=+2871.554403912" watchObservedRunningTime="2026-07-17 10:14:49.017929908 +0000 UTC m=+2871.557128874" Jul 17 10:14:49.871520 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:49.871434 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:49.876243 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:49.876222 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:50.000126 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:50.000090 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:50.000871 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:50.000852 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:14:51.133889 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.133847 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp"] Jul 17 10:14:51.138052 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.138031 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.140820 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.140799 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d4c7ee-serving-cert\"" Jul 17 10:14:51.154103 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.154079 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp"] Jul 17 10:14:51.302719 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.302681 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0988daf-27e4-4e2d-b668-5ddcec7907be-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.302886 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.302749 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0988daf-27e4-4e2d-b668-5ddcec7907be-maas-api-tls\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.302886 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.302836 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dw7p\" (UniqueName: \"kubernetes.io/projected/f0988daf-27e4-4e2d-b668-5ddcec7907be-kube-api-access-8dw7p\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.404095 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.404010 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0988daf-27e4-4e2d-b668-5ddcec7907be-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.404095 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.404068 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0988daf-27e4-4e2d-b668-5ddcec7907be-maas-api-tls\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.404269 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.404100 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dw7p\" (UniqueName: \"kubernetes.io/projected/f0988daf-27e4-4e2d-b668-5ddcec7907be-kube-api-access-8dw7p\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.404692 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.404658 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0988daf-27e4-4e2d-b668-5ddcec7907be-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.406542 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.406522 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0988daf-27e4-4e2d-b668-5ddcec7907be-maas-api-tls\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.412691 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.412671 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dw7p\" (UniqueName: \"kubernetes.io/projected/f0988daf-27e4-4e2d-b668-5ddcec7907be-kube-api-access-8dw7p\") pod \"maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.451963 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.451932 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:51.586126 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.586093 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp"] Jul 17 10:14:51.587051 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:14:51.587024 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0988daf_27e4_4e2d_b668_5ddcec7907be.slice/crio-690c480f0412639c5dcc7ee7435a388b7e6ae361398cee85f914265c3c299bf1 WatchSource:0}: Error finding container 690c480f0412639c5dcc7ee7435a388b7e6ae361398cee85f914265c3c299bf1: Status 404 returned error can't find the container with id 690c480f0412639c5dcc7ee7435a388b7e6ae361398cee85f914265c3c299bf1 Jul 17 10:14:51.588628 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:51.588609 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 10:14:52.012099 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:52.012063 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" event={"ID":"f0988daf-27e4-4e2d-b668-5ddcec7907be","Type":"ContainerStarted","Data":"690c480f0412639c5dcc7ee7435a388b7e6ae361398cee85f914265c3c299bf1"} Jul 17 10:14:53.025736 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:53.025697 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" event={"ID":"f0988daf-27e4-4e2d-b668-5ddcec7907be","Type":"ContainerStarted","Data":"ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a"} Jul 17 10:14:53.026189 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:53.025867 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:53.046939 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:53.046894 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" podStartSLOduration=1.371405583 podStartE2EDuration="2.046876495s" totalImagesPullingTime="675.470912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:14:53.045795862 +0000 UTC m=+2875.584994846" watchObservedRunningTime="2026-07-17 10:14:53.046876495 +0000 UTC m=+2875.586075465" Jul 17 10:14:59.037165 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.037130 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:14:59.326273 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.326191 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl"] Jul 17 10:14:59.331346 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.331328 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.334015 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.333997 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1baf67-serving-cert\"" Jul 17 10:14:59.349152 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.349117 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl"] Jul 17 10:14:59.378145 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.378117 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clqk6\" (UniqueName: \"kubernetes.io/projected/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-kube-api-access-clqk6\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.378302 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.378167 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-openshift-service-ca\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.378302 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.378224 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.479056 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.479020 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clqk6\" (UniqueName: \"kubernetes.io/projected/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-kube-api-access-clqk6\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.479240 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.479092 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-openshift-service-ca\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.479240 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.479159 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.479327 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:14:59.479285 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-serving-cert: secret "maas-api-e2e-api-b-1baf67-serving-cert" not found Jul 17 10:14:59.479376 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:14:59.479346 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls podName:fb6e4bd7-d131-4753-98b3-ffab112f7bc4 nodeName:}" failed. No retries permitted until 2026-07-17 10:14:59.979330425 +0000 UTC m=+2882.518529372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls") pod "maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" (UID: "fb6e4bd7-d131-4753-98b3-ffab112f7bc4") : secret "maas-api-e2e-api-b-1baf67-serving-cert" not found Jul 17 10:14:59.479737 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.479719 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-openshift-service-ca\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.490164 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.490139 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clqk6\" (UniqueName: \"kubernetes.io/projected/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-kube-api-access-clqk6\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.982648 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.982606 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:14:59.985118 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:14:59.985087 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls\") pod \"maas-api-e2e-api-b-1baf67-747dfb57-q2vdl\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:15:00.257714 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:00.257634 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:15:00.409259 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:00.409232 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl"] Jul 17 10:15:00.411252 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:15:00.411222 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6e4bd7_d131_4753_98b3_ffab112f7bc4.slice/crio-feca8686a4b692f3b8e2bdac04dffc193904b78412a68797cf90e85c7934439e WatchSource:0}: Error finding container feca8686a4b692f3b8e2bdac04dffc193904b78412a68797cf90e85c7934439e: Status 404 returned error can't find the container with id feca8686a4b692f3b8e2bdac04dffc193904b78412a68797cf90e85c7934439e Jul 17 10:15:01.083714 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:01.083679 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" event={"ID":"fb6e4bd7-d131-4753-98b3-ffab112f7bc4","Type":"ContainerStarted","Data":"eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db"} Jul 17 10:15:01.083888 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:01.083734 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" event={"ID":"fb6e4bd7-d131-4753-98b3-ffab112f7bc4","Type":"ContainerStarted","Data":"feca8686a4b692f3b8e2bdac04dffc193904b78412a68797cf90e85c7934439e"} Jul 17 10:15:01.083888 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:01.083861 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:15:01.102277 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:01.102226 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" podStartSLOduration=1.633070762 podStartE2EDuration="2.102211316s" totalImagesPullingTime="469.140554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:15:01.09956562 +0000 UTC m=+2883.638764586" watchObservedRunningTime="2026-07-17 10:15:01.102211316 +0000 UTC m=+2883.641410281" Jul 17 10:15:03.095954 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:03.095919 2584 generic.go:356] "Generic (PLEG): container finished" podID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerID="a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba" exitCode=0 Jul 17 10:15:03.096480 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:03.096012 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" event={"ID":"f8a100ee-8398-4d0e-b77d-7dd2a090eb60","Type":"ContainerDied","Data":"a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba"} Jul 17 10:15:03.096545 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:03.096492 2584 scope.go:117] "RemoveContainer" containerID="a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba" Jul 17 10:15:03.097777 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:03.097747 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:15:03.097898 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:03.097830 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:15:03.097898 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:03.097869 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:15:04.103311 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:04.103280 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" event={"ID":"f8a100ee-8398-4d0e-b77d-7dd2a090eb60","Type":"ContainerStarted","Data":"4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9"} Jul 17 10:15:04.871811 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:04.871774 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:15:04.872939 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:04.872908 2584 patch_prober.go:28] interesting pod/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 10:15:04.873060 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:04.872965 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 10:15:05.000205 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:05.000176 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:15:05.872008 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:05.871979 2584 patch_prober.go:28] interesting pod/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 10:15:05.872413 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:05.872030 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 10:15:06.871356 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:06.871318 2584 patch_prober.go:28] interesting pod/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 10:15:06.871576 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:06.871376 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 10:15:07.094172 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:07.094143 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:15:07.871482 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:07.871439 2584 patch_prober.go:28] interesting pod/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 17 10:15:07.871664 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:07.871510 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 17 10:15:08.875648 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:08.875621 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:15:08.876800 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:08.876783 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:15:22.679657 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:22.679617 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp"] Jul 17 10:15:22.680300 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:22.679873 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" podUID="f0988daf-27e4-4e2d-b668-5ddcec7907be" containerName="maas-api" containerID="cri-o://ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a" gracePeriod=30 Jul 17 10:15:22.936940 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:22.936880 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:15:23.003990 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.003959 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dw7p\" (UniqueName: \"kubernetes.io/projected/f0988daf-27e4-4e2d-b668-5ddcec7907be-kube-api-access-8dw7p\") pod \"f0988daf-27e4-4e2d-b668-5ddcec7907be\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " Jul 17 10:15:23.004143 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.004037 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0988daf-27e4-4e2d-b668-5ddcec7907be-maas-api-tls\") pod \"f0988daf-27e4-4e2d-b668-5ddcec7907be\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " Jul 17 10:15:23.004143 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.004084 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0988daf-27e4-4e2d-b668-5ddcec7907be-openshift-service-ca\") pod \"f0988daf-27e4-4e2d-b668-5ddcec7907be\" (UID: \"f0988daf-27e4-4e2d-b668-5ddcec7907be\") " Jul 17 10:15:23.004509 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.004484 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0988daf-27e4-4e2d-b668-5ddcec7907be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0988daf-27e4-4e2d-b668-5ddcec7907be" (UID: "f0988daf-27e4-4e2d-b668-5ddcec7907be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:15:23.005932 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.005912 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0988daf-27e4-4e2d-b668-5ddcec7907be-kube-api-access-8dw7p" (OuterVolumeSpecName: "kube-api-access-8dw7p") pod "f0988daf-27e4-4e2d-b668-5ddcec7907be" (UID: "f0988daf-27e4-4e2d-b668-5ddcec7907be"). InnerVolumeSpecName "kube-api-access-8dw7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:15:23.006090 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.006077 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0988daf-27e4-4e2d-b668-5ddcec7907be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0988daf-27e4-4e2d-b668-5ddcec7907be" (UID: "f0988daf-27e4-4e2d-b668-5ddcec7907be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:15:23.104753 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.104726 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0988daf-27e4-4e2d-b668-5ddcec7907be-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:23.104753 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.104752 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0988daf-27e4-4e2d-b668-5ddcec7907be-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:23.104935 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.104761 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dw7p\" (UniqueName: \"kubernetes.io/projected/f0988daf-27e4-4e2d-b668-5ddcec7907be-kube-api-access-8dw7p\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:23.203409 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.203316 2584 generic.go:356] "Generic (PLEG): container finished" podID="f0988daf-27e4-4e2d-b668-5ddcec7907be" containerID="ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a" exitCode=0 Jul 17 10:15:23.203409 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.203360 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" Jul 17 10:15:23.203574 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.203424 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" event={"ID":"f0988daf-27e4-4e2d-b668-5ddcec7907be","Type":"ContainerDied","Data":"ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a"} Jul 17 10:15:23.203574 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.203477 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp" event={"ID":"f0988daf-27e4-4e2d-b668-5ddcec7907be","Type":"ContainerDied","Data":"690c480f0412639c5dcc7ee7435a388b7e6ae361398cee85f914265c3c299bf1"} Jul 17 10:15:23.203574 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.203503 2584 scope.go:117] "RemoveContainer" containerID="ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a" Jul 17 10:15:23.220023 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.220002 2584 scope.go:117] "RemoveContainer" containerID="ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a" Jul 17 10:15:23.220307 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:15:23.220287 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a\": container with ID starting with ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a not found: ID does not exist" containerID="ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a" Jul 17 10:15:23.220367 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.220315 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a"} err="failed to get container status \"ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a\": rpc error: code = NotFound desc = could not find container \"ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a\": container with ID starting with ed8c6633fdcec60daddd4b9fc730656f3ef108575f7d70f036afe3dacdac748a not found: ID does not exist" Jul 17 10:15:23.231904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.231878 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp"] Jul 17 10:15:23.235685 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.235659 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4c7ee-8667ffb9c7-fs2qp"] Jul 17 10:15:23.999371 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:23.999339 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0988daf-27e4-4e2d-b668-5ddcec7907be" path="/var/lib/kubelet/pods/f0988daf-27e4-4e2d-b668-5ddcec7907be/volumes" Jul 17 10:15:37.586553 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:37.586518 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67"] Jul 17 10:15:37.586933 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:37.586777 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" containerID="cri-o://4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9" gracePeriod=30 Jul 17 10:15:38.878807 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:38.878766 2584 patch_prober.go:28] interesting pod/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 10:15:38.879189 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:38.878848 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 10:15:42.835326 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.835304 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:15:42.996597 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996517 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-socket\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996597 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996550 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-envoy\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996597 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996569 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istiod-ca-cert\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996597 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996588 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-data\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996940 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996621 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-token\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996940 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996694 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-certs\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996940 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996720 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-podinfo\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996940 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996748 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-credential-socket\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996940 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996803 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ldfm\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-kube-api-access-7ldfm\") pod \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\" (UID: \"f8a100ee-8398-4d0e-b77d-7dd2a090eb60\") " Jul 17 10:15:42.996940 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996824 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:15:42.997229 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996953 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:15:42.997229 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.997058 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:15:42.997229 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.997064 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:42.997229 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.996999 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:15:42.997229 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.997097 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:42.997608 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.997571 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:15:42.998759 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.998738 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:15:42.998903 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.998882 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:15:42.999065 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.999043 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-kube-api-access-7ldfm" (OuterVolumeSpecName: "kube-api-access-7ldfm") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "kube-api-access-7ldfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:15:42.999164 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:42.999065 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8a100ee-8398-4d0e-b77d-7dd2a090eb60" (UID: "f8a100ee-8398-4d0e-b77d-7dd2a090eb60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 10:15:43.098708 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.098679 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:43.098708 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.098710 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:43.098945 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.098738 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ldfm\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-kube-api-access-7ldfm\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:43.098945 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.098760 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:43.098945 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.098775 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:43.098945 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.098789 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:43.098945 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.098802 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a100ee-8398-4d0e-b77d-7dd2a090eb60-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:43.318729 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.318645 2584 generic.go:356] "Generic (PLEG): container finished" podID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerID="4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9" exitCode=0 Jul 17 10:15:43.318729 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.318696 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" Jul 17 10:15:43.318729 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.318706 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" event={"ID":"f8a100ee-8398-4d0e-b77d-7dd2a090eb60","Type":"ContainerDied","Data":"4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9"} Jul 17 10:15:43.318985 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.318748 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67" event={"ID":"f8a100ee-8398-4d0e-b77d-7dd2a090eb60","Type":"ContainerDied","Data":"47abcb88a5802eb4defce449ee4822292cc26e37b4c5cdd51fa9e91017eec7d7"} Jul 17 10:15:43.318985 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.318768 2584 scope.go:117] "RemoveContainer" containerID="4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9" Jul 17 10:15:43.328048 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.328027 2584 scope.go:117] "RemoveContainer" containerID="a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba" Jul 17 10:15:43.335812 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.335795 2584 scope.go:117] "RemoveContainer" containerID="4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9" Jul 17 10:15:43.336049 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:15:43.336028 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9\": container with ID starting with 4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9 not found: ID does not exist" containerID="4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9" Jul 17 10:15:43.336088 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.336059 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9"} err="failed to get container status \"4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9\": rpc error: code = NotFound desc = could not find container \"4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9\": container with ID starting with 4467d8d57f7f183144225c1b9e5c73a9ef5743888db4165d2ef5e1be10a5dfc9 not found: ID does not exist" Jul 17 10:15:43.336088 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.336076 2584 scope.go:117] "RemoveContainer" containerID="a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba" Jul 17 10:15:43.336295 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:15:43.336279 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba\": container with ID starting with a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba not found: ID does not exist" containerID="a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba" Jul 17 10:15:43.336330 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.336303 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba"} err="failed to get container status \"a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba\": rpc error: code = NotFound desc = could not find container \"a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba\": container with ID starting with a7c8d4f1930b5aeb14dd42f241281d3cd11ebb1dd943fe62e153cc4895f660ba not found: ID does not exist" Jul 17 10:15:43.345182 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.344045 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67"] Jul 17 10:15:43.347928 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.347909 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d4c7ee-openshift-default-85fbffcdd7-v9m67"] Jul 17 10:15:43.998871 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:43.998830 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" path="/var/lib/kubelet/pods/f8a100ee-8398-4d0e-b77d-7dd2a090eb60/volumes" Jul 17 10:15:48.408731 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.408691 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl"] Jul 17 10:15:48.409156 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.408963 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" podUID="fb6e4bd7-d131-4753-98b3-ffab112f7bc4" containerName="maas-api" containerID="cri-o://eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db" gracePeriod=30 Jul 17 10:15:48.660534 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.660474 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:15:48.745338 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.745312 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clqk6\" (UniqueName: \"kubernetes.io/projected/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-kube-api-access-clqk6\") pod \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " Jul 17 10:15:48.745527 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.745378 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls\") pod \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " Jul 17 10:15:48.745527 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.745440 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-openshift-service-ca\") pod \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\" (UID: \"fb6e4bd7-d131-4753-98b3-ffab112f7bc4\") " Jul 17 10:15:48.745806 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.745785 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb6e4bd7-d131-4753-98b3-ffab112f7bc4" (UID: "fb6e4bd7-d131-4753-98b3-ffab112f7bc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:15:48.747435 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.747408 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb6e4bd7-d131-4753-98b3-ffab112f7bc4" (UID: "fb6e4bd7-d131-4753-98b3-ffab112f7bc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:15:48.747556 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.747531 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-kube-api-access-clqk6" (OuterVolumeSpecName: "kube-api-access-clqk6") pod "fb6e4bd7-d131-4753-98b3-ffab112f7bc4" (UID: "fb6e4bd7-d131-4753-98b3-ffab112f7bc4"). InnerVolumeSpecName "kube-api-access-clqk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:15:48.846012 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.845983 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clqk6\" (UniqueName: \"kubernetes.io/projected/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-kube-api-access-clqk6\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:48.846012 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.846008 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:48.846178 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:48.846019 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb6e4bd7-d131-4753-98b3-ffab112f7bc4-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:15:49.355033 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.354999 2584 generic.go:356] "Generic (PLEG): container finished" podID="fb6e4bd7-d131-4753-98b3-ffab112f7bc4" containerID="eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db" exitCode=0 Jul 17 10:15:49.355197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.355043 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" Jul 17 10:15:49.355197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.355081 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" event={"ID":"fb6e4bd7-d131-4753-98b3-ffab112f7bc4","Type":"ContainerDied","Data":"eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db"} Jul 17 10:15:49.355197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.355134 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl" event={"ID":"fb6e4bd7-d131-4753-98b3-ffab112f7bc4","Type":"ContainerDied","Data":"feca8686a4b692f3b8e2bdac04dffc193904b78412a68797cf90e85c7934439e"} Jul 17 10:15:49.355197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.355158 2584 scope.go:117] "RemoveContainer" containerID="eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db" Jul 17 10:15:49.363927 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.363912 2584 scope.go:117] "RemoveContainer" containerID="eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db" Jul 17 10:15:49.364176 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:15:49.364155 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db\": container with ID starting with eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db not found: ID does not exist" containerID="eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db" Jul 17 10:15:49.364241 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.364187 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db"} err="failed to get container status \"eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db\": rpc error: code = NotFound desc = could not find container \"eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db\": container with ID starting with eb096880c97100eb0a15922e0ee481be2b4c36a6df1979a44a8ede95e13d36db not found: ID does not exist" Jul 17 10:15:49.378463 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.378441 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl"] Jul 17 10:15:49.386418 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.386373 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1baf67-747dfb57-q2vdl"] Jul 17 10:15:49.998486 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:15:49.998458 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb6e4bd7-d131-4753-98b3-ffab112f7bc4" path="/var/lib/kubelet/pods/fb6e4bd7-d131-4753-98b3-ffab112f7bc4/volumes" Jul 17 10:16:04.646499 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646463 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs"] Jul 17 10:16:04.646938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646836 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" Jul 17 10:16:04.646938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646848 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" Jul 17 10:16:04.646938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646857 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0988daf-27e4-4e2d-b668-5ddcec7907be" containerName="maas-api" Jul 17 10:16:04.646938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646864 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0988daf-27e4-4e2d-b668-5ddcec7907be" containerName="maas-api" Jul 17 10:16:04.646938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646874 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb6e4bd7-d131-4753-98b3-ffab112f7bc4" containerName="maas-api" Jul 17 10:16:04.646938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646880 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb6e4bd7-d131-4753-98b3-ffab112f7bc4" containerName="maas-api" Jul 17 10:16:04.646938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646887 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" Jul 17 10:16:04.646938 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646892 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" Jul 17 10:16:04.647197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646957 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb6e4bd7-d131-4753-98b3-ffab112f7bc4" containerName="maas-api" Jul 17 10:16:04.647197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646965 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" Jul 17 10:16:04.647197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646974 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a100ee-8398-4d0e-b77d-7dd2a090eb60" containerName="istio-proxy" Jul 17 10:16:04.647197 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.646980 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0988daf-27e4-4e2d-b668-5ddcec7907be" containerName="maas-api" Jul 17 10:16:04.651653 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.651632 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.654438 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.654417 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ed096b-openshift-default-dockercfg-5l9hr\"" Jul 17 10:16:04.667242 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.662850 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs"] Jul 17 10:16:04.800094 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800061 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-data\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.800094 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800101 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1a23e2-264d-41d5-975e-018ed0db8be2-istiod-ca-cert\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.800311 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800125 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-credential-socket\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.800311 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800241 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-token\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.800311 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800305 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-socket\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.800469 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800321 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-podinfo\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.800469 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800337 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-kube-api-access-jnb9z\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.800469 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800353 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-certs\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.800469 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.800377 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-envoy\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.901535 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901450 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1a23e2-264d-41d5-975e-018ed0db8be2-istiod-ca-cert\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.901535 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901496 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-credential-socket\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.901772 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901575 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-token\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.901772 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901640 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-socket\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.901772 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901664 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-podinfo\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.901772 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901685 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-kube-api-access-jnb9z\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.901772 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901709 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-certs\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.901772 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901745 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-envoy\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.902108 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901780 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-data\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.902108 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.901962 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-credential-socket\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.902108 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.902022 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-socket\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.902108 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.902048 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-certs\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.902280 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.902135 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-data\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.902280 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.902231 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1a23e2-264d-41d5-975e-018ed0db8be2-istiod-ca-cert\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.903970 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.903947 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-envoy\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.904091 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.904074 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-podinfo\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.916475 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.916453 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-token\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.916928 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.916905 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-kube-api-access-jnb9z\") pod \"e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:04.969833 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:04.969786 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:05.108936 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:05.108908 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs"] Jul 17 10:16:05.110945 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:16:05.110921 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1a23e2_264d_41d5_975e_018ed0db8be2.slice/crio-c99d8dc89e42e07a23127f593576bc69f82fce3b3b1da1477b1a2828d62cb5d4 WatchSource:0}: Error finding container c99d8dc89e42e07a23127f593576bc69f82fce3b3b1da1477b1a2828d62cb5d4: Status 404 returned error can't find the container with id c99d8dc89e42e07a23127f593576bc69f82fce3b3b1da1477b1a2828d62cb5d4 Jul 17 10:16:05.112926 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:05.112897 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:16:05.112979 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:05.112960 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:16:05.113014 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:05.112988 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:16:05.442279 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:05.442191 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" event={"ID":"5c1a23e2-264d-41d5-975e-018ed0db8be2","Type":"ContainerStarted","Data":"adc73043ce00fb8eb7e0141908cc3fc3375d5750e206a32587a99e3d97d1fc74"} Jul 17 10:16:05.442279 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:05.442245 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" event={"ID":"5c1a23e2-264d-41d5-975e-018ed0db8be2","Type":"ContainerStarted","Data":"c99d8dc89e42e07a23127f593576bc69f82fce3b3b1da1477b1a2828d62cb5d4"} Jul 17 10:16:05.478154 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:05.478108 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" podStartSLOduration=1.478094765 podStartE2EDuration="1.478094765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:16:05.475793816 +0000 UTC m=+2948.014992781" watchObservedRunningTime="2026-07-17 10:16:05.478094765 +0000 UTC m=+2948.017293730" Jul 17 10:16:05.970348 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:05.970309 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:06.974942 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:06.974909 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:07.459267 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.459239 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:07.460128 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.460112 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:07.560446 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.560409 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz"] Jul 17 10:16:07.563830 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.563811 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:07.566235 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.566215 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ed096b-serving-cert\"" Jul 17 10:16:07.574254 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.574227 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz"] Jul 17 10:16:07.628946 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.628904 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:07.629123 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.628966 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:07.629123 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.629007 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkxmm\" (UniqueName: \"kubernetes.io/projected/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-kube-api-access-bkxmm\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:07.730577 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.730485 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:07.730577 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.730535 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkxmm\" (UniqueName: \"kubernetes.io/projected/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-kube-api-access-bkxmm\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:07.730776 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.730606 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:07.730776 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:16:07.730647 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-serving-cert: secret "maas-api-e2e-shared-a-ed096b-serving-cert" not found Jul 17 10:16:07.730776 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:16:07.730709 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls podName:dce6d7c0-daa2-4b32-a651-f2c1c94e94d3 nodeName:}" failed. No retries permitted until 2026-07-17 10:16:08.230691601 +0000 UTC m=+2950.769890545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls") pod "maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" (UID: "dce6d7c0-daa2-4b32-a651-f2c1c94e94d3") : secret "maas-api-e2e-shared-a-ed096b-serving-cert" not found Jul 17 10:16:07.731248 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.731230 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:07.742103 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:07.742077 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkxmm\" (UniqueName: \"kubernetes.io/projected/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-kube-api-access-bkxmm\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:08.234320 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:08.234281 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:08.236709 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:08.236678 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:08.476072 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:08.476040 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:08.608775 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:08.608738 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz"] Jul 17 10:16:08.610072 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:16:08.610048 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce6d7c0_daa2_4b32_a651_f2c1c94e94d3.slice/crio-0842816b85b5903e93fd5ae2f6ac3214d3dd383162c037f4cd7c20bbfae77f0d WatchSource:0}: Error finding container 0842816b85b5903e93fd5ae2f6ac3214d3dd383162c037f4cd7c20bbfae77f0d: Status 404 returned error can't find the container with id 0842816b85b5903e93fd5ae2f6ac3214d3dd383162c037f4cd7c20bbfae77f0d Jul 17 10:16:09.470841 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:09.470805 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" event={"ID":"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3","Type":"ContainerStarted","Data":"eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e"} Jul 17 10:16:09.471269 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:09.470871 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" event={"ID":"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3","Type":"ContainerStarted","Data":"0842816b85b5903e93fd5ae2f6ac3214d3dd383162c037f4cd7c20bbfae77f0d"} Jul 17 10:16:09.471269 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:09.471012 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:09.487669 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:09.487620 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" podStartSLOduration=2.038827476 podStartE2EDuration="2.487604607s" totalImagesPullingTime="448.777131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:16:09.486331204 +0000 UTC m=+2952.025530171" watchObservedRunningTime="2026-07-17 10:16:09.487604607 +0000 UTC m=+2952.026803573" Jul 17 10:16:15.481736 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:15.481705 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:16:15.820242 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:15.820160 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r"] Jul 17 10:16:15.823874 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:15.823855 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:15.826348 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:15.826327 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-af80d9-serving-cert\"" Jul 17 10:16:15.863420 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:15.860474 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r"] Jul 17 10:16:15.911114 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:15.911074 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc99v\" (UniqueName: \"kubernetes.io/projected/21bebf41-2300-413c-bef5-5af4800228eb-kube-api-access-fc99v\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:15.911293 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:15.911154 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bebf41-2300-413c-bef5-5af4800228eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:15.911293 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:15.911195 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.012153 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.012121 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bebf41-2300-413c-bef5-5af4800228eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.012326 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.012187 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.012326 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.012228 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc99v\" (UniqueName: \"kubernetes.io/projected/21bebf41-2300-413c-bef5-5af4800228eb-kube-api-access-fc99v\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.012430 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:16:16.012373 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-serving-cert: secret "maas-api-e2e-shared-b-af80d9-serving-cert" not found Jul 17 10:16:16.012510 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:16:16.012496 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls podName:21bebf41-2300-413c-bef5-5af4800228eb nodeName:}" failed. No retries permitted until 2026-07-17 10:16:16.512471063 +0000 UTC m=+2959.051670029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls") pod "maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" (UID: "21bebf41-2300-413c-bef5-5af4800228eb") : secret "maas-api-e2e-shared-b-af80d9-serving-cert" not found Jul 17 10:16:16.012768 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.012751 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bebf41-2300-413c-bef5-5af4800228eb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.020430 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.020386 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc99v\" (UniqueName: \"kubernetes.io/projected/21bebf41-2300-413c-bef5-5af4800228eb-kube-api-access-fc99v\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.517106 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.517077 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.519468 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.519446 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls\") pod \"maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.736484 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.736439 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:16.916470 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:16.914482 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r"] Jul 17 10:16:16.919454 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:16:16.919419 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bebf41_2300_413c_bef5_5af4800228eb.slice/crio-837a15b06fb2eae03abfad5d00a6e97eea14ba25ba002568dfbaa4d147deb226 WatchSource:0}: Error finding container 837a15b06fb2eae03abfad5d00a6e97eea14ba25ba002568dfbaa4d147deb226: Status 404 returned error can't find the container with id 837a15b06fb2eae03abfad5d00a6e97eea14ba25ba002568dfbaa4d147deb226 Jul 17 10:16:17.519459 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:17.519422 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" event={"ID":"21bebf41-2300-413c-bef5-5af4800228eb","Type":"ContainerStarted","Data":"d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80"} Jul 17 10:16:17.519862 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:17.519481 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" event={"ID":"21bebf41-2300-413c-bef5-5af4800228eb","Type":"ContainerStarted","Data":"837a15b06fb2eae03abfad5d00a6e97eea14ba25ba002568dfbaa4d147deb226"} Jul 17 10:16:17.519862 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:17.519537 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:17.544711 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:17.544591 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" podStartSLOduration=2.175717807 podStartE2EDuration="2.544550168s" totalImagesPullingTime="368.832361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 10:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:16:17.540348134 +0000 UTC m=+2960.079547101" watchObservedRunningTime="2026-07-17 10:16:17.544550168 +0000 UTC m=+2960.083749135" Jul 17 10:16:21.544673 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:21.544578 2584 generic.go:356] "Generic (PLEG): container finished" podID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerID="adc73043ce00fb8eb7e0141908cc3fc3375d5750e206a32587a99e3d97d1fc74" exitCode=0 Jul 17 10:16:21.544673 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:21.544660 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" event={"ID":"5c1a23e2-264d-41d5-975e-018ed0db8be2","Type":"ContainerDied","Data":"adc73043ce00fb8eb7e0141908cc3fc3375d5750e206a32587a99e3d97d1fc74"} Jul 17 10:16:21.545118 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:21.545103 2584 scope.go:117] "RemoveContainer" containerID="adc73043ce00fb8eb7e0141908cc3fc3375d5750e206a32587a99e3d97d1fc74" Jul 17 10:16:21.546101 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:21.546078 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:16:21.546162 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:21.546153 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:16:21.546199 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:21.546183 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 17 10:16:22.459464 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:22.459431 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:22.555778 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:22.555736 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" event={"ID":"5c1a23e2-264d-41d5-975e-018ed0db8be2","Type":"ContainerStarted","Data":"b93cb95e637034ee7ca921a18afe3293848e711e7dd2fda295f79e785882c4f9"} Jul 17 10:16:22.970586 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:22.970547 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:22.971763 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:22.971737 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 17 10:16:22.971871 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:22.971786 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 17 10:16:23.531928 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:23.531893 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:16:23.971000 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:23.970962 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 17 10:16:23.971462 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:23.971038 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 17 10:16:24.970839 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:24.970796 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 17 10:16:24.971031 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:24.970869 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 17 10:16:25.971198 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:25.971157 2584 patch_prober.go:28] interesting pod/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 17 10:16:25.971640 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:25.971225 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 17 10:16:26.974796 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:26.974759 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:26.975169 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:26.975117 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:26.975841 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:26.975821 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:16:58.211449 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:58.211419 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:16:58.220356 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:16:58.220332 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:21:58.251081 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:21:58.251051 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:21:58.258634 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:21:58.258605 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:23:02.285809 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.285745 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz"] Jul 17 10:23:02.286416 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.286060 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" podUID="dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" containerName="maas-api" containerID="cri-o://eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e" gracePeriod=30 Jul 17 10:23:02.544951 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.544882 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:23:02.684180 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.684144 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-openshift-service-ca\") pod \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " Jul 17 10:23:02.684379 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.684262 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls\") pod \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " Jul 17 10:23:02.684379 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.684304 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkxmm\" (UniqueName: \"kubernetes.io/projected/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-kube-api-access-bkxmm\") pod \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\" (UID: \"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3\") " Jul 17 10:23:02.684644 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.684612 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" (UID: "dce6d7c0-daa2-4b32-a651-f2c1c94e94d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:23:02.686485 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.686455 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-kube-api-access-bkxmm" (OuterVolumeSpecName: "kube-api-access-bkxmm") pod "dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" (UID: "dce6d7c0-daa2-4b32-a651-f2c1c94e94d3"). InnerVolumeSpecName "kube-api-access-bkxmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:23:02.686485 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.686473 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" (UID: "dce6d7c0-daa2-4b32-a651-f2c1c94e94d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:23:02.721727 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.721696 2584 generic.go:356] "Generic (PLEG): container finished" podID="dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" containerID="eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e" exitCode=0 Jul 17 10:23:02.721896 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.721742 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" Jul 17 10:23:02.721896 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.721754 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" event={"ID":"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3","Type":"ContainerDied","Data":"eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e"} Jul 17 10:23:02.721896 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.721795 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz" event={"ID":"dce6d7c0-daa2-4b32-a651-f2c1c94e94d3","Type":"ContainerDied","Data":"0842816b85b5903e93fd5ae2f6ac3214d3dd383162c037f4cd7c20bbfae77f0d"} Jul 17 10:23:02.721896 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.721815 2584 scope.go:117] "RemoveContainer" containerID="eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e" Jul 17 10:23:02.732186 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.732166 2584 scope.go:117] "RemoveContainer" containerID="eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e" Jul 17 10:23:02.732464 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:23:02.732441 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e\": container with ID starting with eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e not found: ID does not exist" containerID="eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e" Jul 17 10:23:02.732535 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.732473 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e"} err="failed to get container status \"eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e\": rpc error: code = NotFound desc = could not find container \"eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e\": container with ID starting with eee776802aa4ab6dd07eff0df3106682cdaa2f50defa80269fedb8fa13d27b5e not found: ID does not exist" Jul 17 10:23:02.745945 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.745915 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz"] Jul 17 10:23:02.751749 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.751724 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed096b-7464c8445b-5qsmz"] Jul 17 10:23:02.784956 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.784926 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:02.784956 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.784953 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkxmm\" (UniqueName: \"kubernetes.io/projected/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-kube-api-access-bkxmm\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:02.785181 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:02.784964 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:03.998962 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:03.998925 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" path="/var/lib/kubelet/pods/dce6d7c0-daa2-4b32-a651-f2c1c94e94d3/volumes" Jul 17 10:23:17.571134 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:17.571100 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs"] Jul 17 10:23:17.571640 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:17.571320 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" containerID="cri-o://b93cb95e637034ee7ca921a18afe3293848e711e7dd2fda295f79e785882c4f9" gracePeriod=30 Jul 17 10:23:22.839175 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.839133 2584 generic.go:356] "Generic (PLEG): container finished" podID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerID="b93cb95e637034ee7ca921a18afe3293848e711e7dd2fda295f79e785882c4f9" exitCode=0 Jul 17 10:23:22.839604 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.839178 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" event={"ID":"5c1a23e2-264d-41d5-975e-018ed0db8be2","Type":"ContainerDied","Data":"b93cb95e637034ee7ca921a18afe3293848e711e7dd2fda295f79e785882c4f9"} Jul 17 10:23:22.839604 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.839243 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" event={"ID":"5c1a23e2-264d-41d5-975e-018ed0db8be2","Type":"ContainerDied","Data":"c99d8dc89e42e07a23127f593576bc69f82fce3b3b1da1477b1a2828d62cb5d4"} Jul 17 10:23:22.839604 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.839259 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c99d8dc89e42e07a23127f593576bc69f82fce3b3b1da1477b1a2828d62cb5d4" Jul 17 10:23:22.839604 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.839273 2584 scope.go:117] "RemoveContainer" containerID="adc73043ce00fb8eb7e0141908cc3fc3375d5750e206a32587a99e3d97d1fc74" Jul 17 10:23:22.842457 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.842437 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:23:22.981637 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981603 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-certs\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.981637 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981649 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-token\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.981914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981708 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-kube-api-access-jnb9z\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.981914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981743 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-socket\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.981914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981776 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-credential-socket\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.981914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981804 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-podinfo\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.981914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981830 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-envoy\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.981914 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981892 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-data\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.982234 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.981947 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:23:22.982234 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982087 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1a23e2-264d-41d5-975e-018ed0db8be2-istiod-ca-cert\") pod \"5c1a23e2-264d-41d5-975e-018ed0db8be2\" (UID: \"5c1a23e2-264d-41d5-975e-018ed0db8be2\") " Jul 17 10:23:22.982234 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982127 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:23:22.982234 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982138 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:23:22.982460 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982439 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c1a23e2-264d-41d5-975e-018ed0db8be2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:23:22.982556 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982539 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:22.982613 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982559 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-credential-socket\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:22.982613 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982569 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c1a23e2-264d-41d5-975e-018ed0db8be2-istiod-ca-cert\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:22.982613 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982577 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-workload-certs\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:22.982998 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.982974 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:23:22.984052 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.984022 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:23:22.984186 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.984164 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 10:23:22.984256 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.984237 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 10:23:22.984435 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:22.984418 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-kube-api-access-jnb9z" (OuterVolumeSpecName: "kube-api-access-jnb9z") pod "5c1a23e2-264d-41d5-975e-018ed0db8be2" (UID: "5c1a23e2-264d-41d5-975e-018ed0db8be2"). InnerVolumeSpecName "kube-api-access-jnb9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:23:23.083519 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:23.083480 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-podinfo\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:23.083519 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:23.083512 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-envoy\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:23.083519 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:23.083522 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-data\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:23.083519 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:23.083529 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-istio-token\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:23.083788 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:23.083537 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnb9z\" (UniqueName: \"kubernetes.io/projected/5c1a23e2-264d-41d5-975e-018ed0db8be2-kube-api-access-jnb9z\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:23.845834 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:23.845804 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs" Jul 17 10:23:23.904069 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:23.904035 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs"] Jul 17 10:23:23.907240 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:23.907217 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ed096b-openshift-default-6cff89f8cf-99tqs"] Jul 17 10:23:24.000115 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:24.000077 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" path="/var/lib/kubelet/pods/5c1a23e2-264d-41d5-975e-018ed0db8be2/volumes" Jul 17 10:23:28.392504 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.392471 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r"] Jul 17 10:23:28.397427 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.393154 2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" podUID="21bebf41-2300-413c-bef5-5af4800228eb" containerName="maas-api" containerID="cri-o://d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80" gracePeriod=30 Jul 17 10:23:28.526254 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.526205 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" podUID="21bebf41-2300-413c-bef5-5af4800228eb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.58:8443/health\": dial tcp 10.134.0.58:8443: connect: connection refused" Jul 17 10:23:28.644252 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.644184 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:23:28.728455 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.728421 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls\") pod \"21bebf41-2300-413c-bef5-5af4800228eb\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " Jul 17 10:23:28.728668 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.728553 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc99v\" (UniqueName: \"kubernetes.io/projected/21bebf41-2300-413c-bef5-5af4800228eb-kube-api-access-fc99v\") pod \"21bebf41-2300-413c-bef5-5af4800228eb\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " Jul 17 10:23:28.728668 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.728602 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bebf41-2300-413c-bef5-5af4800228eb-openshift-service-ca\") pod \"21bebf41-2300-413c-bef5-5af4800228eb\" (UID: \"21bebf41-2300-413c-bef5-5af4800228eb\") " Jul 17 10:23:28.729006 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.728973 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bebf41-2300-413c-bef5-5af4800228eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "21bebf41-2300-413c-bef5-5af4800228eb" (UID: "21bebf41-2300-413c-bef5-5af4800228eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 10:23:28.730589 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.730565 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bebf41-2300-413c-bef5-5af4800228eb-kube-api-access-fc99v" (OuterVolumeSpecName: "kube-api-access-fc99v") pod "21bebf41-2300-413c-bef5-5af4800228eb" (UID: "21bebf41-2300-413c-bef5-5af4800228eb"). InnerVolumeSpecName "kube-api-access-fc99v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 10:23:28.730675 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.730595 2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "21bebf41-2300-413c-bef5-5af4800228eb" (UID: "21bebf41-2300-413c-bef5-5af4800228eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 10:23:28.829714 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.829668 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc99v\" (UniqueName: \"kubernetes.io/projected/21bebf41-2300-413c-bef5-5af4800228eb-kube-api-access-fc99v\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:28.829714 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.829704 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21bebf41-2300-413c-bef5-5af4800228eb-openshift-service-ca\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:28.829714 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.829718 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21bebf41-2300-413c-bef5-5af4800228eb-maas-api-tls\") on node \"ip-10-0-133-236.ec2.internal\" DevicePath \"\"" Jul 17 10:23:28.877451 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.877418 2584 generic.go:356] "Generic (PLEG): container finished" podID="21bebf41-2300-413c-bef5-5af4800228eb" containerID="d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80" exitCode=0 Jul 17 10:23:28.877451 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.877449 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" Jul 17 10:23:28.877637 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.877478 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" event={"ID":"21bebf41-2300-413c-bef5-5af4800228eb","Type":"ContainerDied","Data":"d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80"} Jul 17 10:23:28.877637 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.877545 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r" event={"ID":"21bebf41-2300-413c-bef5-5af4800228eb","Type":"ContainerDied","Data":"837a15b06fb2eae03abfad5d00a6e97eea14ba25ba002568dfbaa4d147deb226"} Jul 17 10:23:28.877637 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.877566 2584 scope.go:117] "RemoveContainer" containerID="d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80" Jul 17 10:23:28.886796 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.886777 2584 scope.go:117] "RemoveContainer" containerID="d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80" Jul 17 10:23:28.887026 ip-10-0-133-236 kubenswrapper[2584]: E0717 10:23:28.887005 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80\": container with ID starting with d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80 not found: ID does not exist" containerID="d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80" Jul 17 10:23:28.887066 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.887036 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80"} err="failed to get container status \"d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80\": rpc error: code = NotFound desc = could not find container \"d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80\": container with ID starting with d8b9326719f48bd2d5cd708d6b1eef61064d4800e640046c3ee341ce0c06de80 not found: ID does not exist" Jul 17 10:23:28.900242 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.900186 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r"] Jul 17 10:23:28.903581 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:28.903558 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-af80d9-7cdf6bd44f-4sr9r"] Jul 17 10:23:29.998999 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:29.998964 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21bebf41-2300-413c-bef5-5af4800228eb" path="/var/lib/kubelet/pods/21bebf41-2300-413c-bef5-5af4800228eb/volumes" Jul 17 10:23:51.041868 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:51.041797 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5566fdf9-4ncqd_ce9315d2-e289-412c-b70b-3fb424690668/copy-manifests/0.log" Jul 17 10:23:51.050065 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:51.050041 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5566fdf9-4ncqd_ce9315d2-e289-412c-b70b-3fb424690668/manager/0.log" Jul 17 10:23:51.600723 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:51.600694 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6957c666b-jklr9_fcd32724-ea72-48a2-ab3e-239e7345db38/maas-api/0.log" Jul 17 10:23:54.106531 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.106499 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd_32491371-c26f-4d72-ab8a-308769d06bcf/extract/0.log" Jul 17 10:23:54.113434 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.113411 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd_32491371-c26f-4d72-ab8a-308769d06bcf/util/0.log" Jul 17 10:23:54.123280 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.123262 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd_32491371-c26f-4d72-ab8a-308769d06bcf/pull/0.log" Jul 17 10:23:54.241925 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.241893 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26_a26f9c86-140d-454d-9c10-9acd72107ac0/util/0.log" Jul 17 10:23:54.248818 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.248789 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26_a26f9c86-140d-454d-9c10-9acd72107ac0/pull/0.log" Jul 17 10:23:54.255574 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.255553 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26_a26f9c86-140d-454d-9c10-9acd72107ac0/extract/0.log" Jul 17 10:23:54.370137 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.370064 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j_8257a986-e99b-4089-8b45-2b41e0340103/util/0.log" Jul 17 10:23:54.376654 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.376631 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j_8257a986-e99b-4089-8b45-2b41e0340103/pull/0.log" Jul 17 10:23:54.384973 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.384953 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j_8257a986-e99b-4089-8b45-2b41e0340103/extract/0.log" Jul 17 10:23:54.496985 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.496958 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259_937498cf-9879-45b0-92ca-3c106854f506/util/0.log" Jul 17 10:23:54.503389 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.503367 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259_937498cf-9879-45b0-92ca-3c106854f506/pull/0.log" Jul 17 10:23:54.509526 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:54.509506 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259_937498cf-9879-45b0-92ca-3c106854f506/extract/0.log" Jul 17 10:23:55.906067 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:55.906042 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ngb4g_82b2d768-3b69-4816-9ac3-1ec68ded2216/istio-proxy/0.log" Jul 17 10:23:56.038260 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:56.038231 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m8966_a6e951b2-4383-4adf-9e5d-0291652188a3/discovery/0.log" Jul 17 10:23:57.311943 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:57.311914 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2cvvj_aa31d65e-8aa0-4dc0-a348-e57adf5e2f18/main/0.log" Jul 17 10:23:57.564801 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:57.564709 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbc65ls_0387d70e-160c-456c-b149-0453d439bb01/main/0.log" Jul 17 10:23:57.685502 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:57.685468 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gtm4t_a674834a-dedc-4f56-a554-557dabad2233/main/0.log" Jul 17 10:23:58.379102 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:23:58.379072 2584 scope.go:117] "RemoveContainer" containerID="b93cb95e637034ee7ca921a18afe3293848e711e7dd2fda295f79e785882c4f9" Jul 17 10:24:07.482531 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:07.482500 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pkvmq_ff2f29e6-ee5e-433c-a625-30090f8f5d9c/global-pull-secret-syncer/0.log" Jul 17 10:24:07.709010 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:07.708982 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sn496_5d99751e-24ae-4349-ab23-b4ce2592a164/konnectivity-agent/0.log" Jul 17 10:24:07.732185 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:07.732162 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-236.ec2.internal_b51292bad5f58bba29f3717a67222f17/haproxy/0.log" Jul 17 10:24:12.548251 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.548218 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd_32491371-c26f-4d72-ab8a-308769d06bcf/extract/0.log" Jul 17 10:24:12.575108 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.575085 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd_32491371-c26f-4d72-ab8a-308769d06bcf/util/0.log" Jul 17 10:24:12.597915 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.597890 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zgkxd_32491371-c26f-4d72-ab8a-308769d06bcf/pull/0.log" Jul 17 10:24:12.626740 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.626710 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26_a26f9c86-140d-454d-9c10-9acd72107ac0/extract/0.log" Jul 17 10:24:12.654336 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.654314 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26_a26f9c86-140d-454d-9c10-9acd72107ac0/util/0.log" Jul 17 10:24:12.685098 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.685076 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nnj26_a26f9c86-140d-454d-9c10-9acd72107ac0/pull/0.log" Jul 17 10:24:12.726885 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.726860 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j_8257a986-e99b-4089-8b45-2b41e0340103/extract/0.log" Jul 17 10:24:12.753890 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.753871 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j_8257a986-e99b-4089-8b45-2b41e0340103/util/0.log" Jul 17 10:24:12.779796 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.779769 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73vmc6j_8257a986-e99b-4089-8b45-2b41e0340103/pull/0.log" Jul 17 10:24:12.808620 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.808561 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259_937498cf-9879-45b0-92ca-3c106854f506/extract/0.log" Jul 17 10:24:12.841275 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.841257 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259_937498cf-9879-45b0-92ca-3c106854f506/util/0.log" Jul 17 10:24:12.864657 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:12.864631 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1hp259_937498cf-9879-45b0-92ca-3c106854f506/pull/0.log" Jul 17 10:24:15.292418 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.292312 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6fe5847-5268-43e8-9d78-1f59b27f57d8/alertmanager/0.log" Jul 17 10:24:15.319974 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.319938 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6fe5847-5268-43e8-9d78-1f59b27f57d8/config-reloader/0.log" Jul 17 10:24:15.348736 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.348717 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6fe5847-5268-43e8-9d78-1f59b27f57d8/kube-rbac-proxy-web/0.log" Jul 17 10:24:15.372988 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.372972 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6fe5847-5268-43e8-9d78-1f59b27f57d8/kube-rbac-proxy/0.log" Jul 17 10:24:15.400227 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.400209 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6fe5847-5268-43e8-9d78-1f59b27f57d8/kube-rbac-proxy-metric/0.log" Jul 17 10:24:15.424507 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.424488 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6fe5847-5268-43e8-9d78-1f59b27f57d8/prom-label-proxy/0.log" Jul 17 10:24:15.449667 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.449645 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6fe5847-5268-43e8-9d78-1f59b27f57d8/init-config-reloader/0.log" Jul 17 10:24:15.502923 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.502893 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-svxdt_1ed006d9-ebc3-4e86-b54a-4e28aa293ba9/cluster-monitoring-operator/0.log" Jul 17 10:24:15.530375 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.530354 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vsccb_251db854-1393-40ee-8cd5-0469a9ea752e/kube-state-metrics/0.log" Jul 17 10:24:15.553271 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.553212 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vsccb_251db854-1393-40ee-8cd5-0469a9ea752e/kube-rbac-proxy-main/0.log" Jul 17 10:24:15.585927 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.585909 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vsccb_251db854-1393-40ee-8cd5-0469a9ea752e/kube-rbac-proxy-self/0.log" Jul 17 10:24:15.775200 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.775170 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2jz4_9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7/node-exporter/0.log" Jul 17 10:24:15.796756 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.796731 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2jz4_9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7/kube-rbac-proxy/0.log" Jul 17 10:24:15.819468 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:15.819387 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2jz4_9ef4032e-c364-4ac6-b6a0-fbf1e499d2d7/init-textfile/0.log" Jul 17 10:24:16.390115 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:16.390073 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b5884db65-tjrzh_e1c089c0-767a-407d-b788-7fde202cba7d/thanos-query/0.log" Jul 17 10:24:16.412422 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:16.412381 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b5884db65-tjrzh_e1c089c0-767a-407d-b788-7fde202cba7d/kube-rbac-proxy-web/0.log" Jul 17 10:24:16.445792 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:16.445769 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b5884db65-tjrzh_e1c089c0-767a-407d-b788-7fde202cba7d/kube-rbac-proxy/0.log" Jul 17 10:24:16.468656 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:16.468638 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b5884db65-tjrzh_e1c089c0-767a-407d-b788-7fde202cba7d/prom-label-proxy/0.log" Jul 17 10:24:16.489703 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:16.489628 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b5884db65-tjrzh_e1c089c0-767a-407d-b788-7fde202cba7d/kube-rbac-proxy-rules/0.log" Jul 17 10:24:16.515306 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:16.515286 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b5884db65-tjrzh_e1c089c0-767a-407d-b788-7fde202cba7d/kube-rbac-proxy-metrics/0.log" Jul 17 10:24:18.039324 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039295 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf"] Jul 17 10:24:18.039788 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039753 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" containerName="maas-api" Jul 17 10:24:18.039788 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039768 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" containerName="maas-api" Jul 17 10:24:18.039788 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039781 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" Jul 17 10:24:18.039788 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039789 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" Jul 17 10:24:18.040002 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039835 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21bebf41-2300-413c-bef5-5af4800228eb" containerName="maas-api" Jul 17 10:24:18.040002 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039843 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="21bebf41-2300-413c-bef5-5af4800228eb" containerName="maas-api" Jul 17 10:24:18.040002 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039903 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="21bebf41-2300-413c-bef5-5af4800228eb" containerName="maas-api" Jul 17 10:24:18.040002 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039911 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce6d7c0-daa2-4b32-a651-f2c1c94e94d3" containerName="maas-api" Jul 17 10:24:18.040002 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039919 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" Jul 17 10:24:18.040002 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.039927 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c1a23e2-264d-41d5-975e-018ed0db8be2" containerName="istio-proxy" Jul 17 10:24:18.043223 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.043206 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.045809 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.045790 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5lvm5\"/\"openshift-service-ca.crt\"" Jul 17 10:24:18.045809 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.045805 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5lvm5\"/\"default-dockercfg-vqkh2\"" Jul 17 10:24:18.046852 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.046838 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5lvm5\"/\"kube-root-ca.crt\"" Jul 17 10:24:18.053559 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.053536 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf"] Jul 17 10:24:18.073886 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.073860 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-sys\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.074016 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.073898 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-lib-modules\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.074016 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.073958 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q66dz\" (UniqueName: \"kubernetes.io/projected/82ceb1d8-3a72-40a1-92e8-08198c43e0db-kube-api-access-q66dz\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.074156 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.074040 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-podres\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.074156 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.074107 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-proc\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.174946 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.174915 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-podres\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.175116 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.174964 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-proc\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.175116 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.174995 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-sys\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.175116 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.175013 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-lib-modules\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.175116 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.175040 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q66dz\" (UniqueName: \"kubernetes.io/projected/82ceb1d8-3a72-40a1-92e8-08198c43e0db-kube-api-access-q66dz\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.175116 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.175107 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-sys\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.175324 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.175108 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-podres\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.175324 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.175138 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-proc\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.175324 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.175161 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/82ceb1d8-3a72-40a1-92e8-08198c43e0db-lib-modules\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.184287 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.184259 2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q66dz\" (UniqueName: \"kubernetes.io/projected/82ceb1d8-3a72-40a1-92e8-08198c43e0db-kube-api-access-q66dz\") pod \"perf-node-gather-daemonset-7mdsf\" (UID: \"82ceb1d8-3a72-40a1-92e8-08198c43e0db\") " pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.278115 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.278094 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/1.log" Jul 17 10:24:18.286742 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.286720 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-4l45k_8de78fc2-54b3-498b-bd7f-4d38b3a23bc7/console-operator/2.log" Jul 17 10:24:18.353951 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.353930 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:18.483303 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:18.483281 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf"] Jul 17 10:24:18.485163 ip-10-0-133-236 kubenswrapper[2584]: W0717 10:24:18.485136 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod82ceb1d8_3a72_40a1_92e8_08198c43e0db.slice/crio-3f765759c6c8583828d10e67cd8ab8140012822c159f62d7df709c22eac7577c WatchSource:0}: Error finding container 3f765759c6c8583828d10e67cd8ab8140012822c159f62d7df709c22eac7577c: Status 404 returned error can't find the container with id 3f765759c6c8583828d10e67cd8ab8140012822c159f62d7df709c22eac7577c Jul 17 10:24:19.132854 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:19.132820 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" event={"ID":"82ceb1d8-3a72-40a1-92e8-08198c43e0db","Type":"ContainerStarted","Data":"3ff7bafe6e756220ca1a0feb90aee40323136dc4eac1003582ac4544f6f01047"} Jul 17 10:24:19.133276 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:19.132871 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" event={"ID":"82ceb1d8-3a72-40a1-92e8-08198c43e0db","Type":"ContainerStarted","Data":"3f765759c6c8583828d10e67cd8ab8140012822c159f62d7df709c22eac7577c"} Jul 17 10:24:19.133276 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:19.132909 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:19.153837 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:19.153792 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" podStartSLOduration=1.153779552 podStartE2EDuration="1.153779552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 10:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 10:24:19.152450472 +0000 UTC m=+3441.691649439" watchObservedRunningTime="2026-07-17 10:24:19.153779552 +0000 UTC m=+3441.692978519" Jul 17 10:24:20.460949 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:20.460904 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6xm7j_26bf1378-169a-44ee-b4a6-3b8116c5cd52/dns/0.log" Jul 17 10:24:20.483904 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:20.483877 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6xm7j_26bf1378-169a-44ee-b4a6-3b8116c5cd52/kube-rbac-proxy/0.log" Jul 17 10:24:20.665415 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:20.665383 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zvws7_eb10dd55-281e-4ebf-ac8f-d79c9e6e55e7/dns-node-resolver/0.log" Jul 17 10:24:21.204447 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:21.204384 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7675b5767f-wp4r5_9ce68b72-471e-40be-a4cf-0c493ccfcca7/registry/0.log" Jul 17 10:24:21.284825 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:21.284796 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hkg55_5ab356ac-a51c-44dc-b850-1a0f814a0b0f/node-ca/0.log" Jul 17 10:24:22.258130 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:22.258104 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ngb4g_82b2d768-3b69-4816-9ac3-1ec68ded2216/istio-proxy/0.log" Jul 17 10:24:22.463822 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:22.463792 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m8966_a6e951b2-4383-4adf-9e5d-0291652188a3/discovery/0.log" Jul 17 10:24:23.402858 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:23.402833 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q545l_8d9a56e4-3df7-4cda-befd-d758b51cb63c/serve-healthcheck-canary/0.log" Jul 17 10:24:23.923844 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:23.923814 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kb2fs_b8c24f15-a691-4a51-a970-a00697042f6e/insights-operator/0.log" Jul 17 10:24:23.925422 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:23.925385 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-kb2fs_b8c24f15-a691-4a51-a970-a00697042f6e/insights-operator/1.log" Jul 17 10:24:23.949603 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:23.949576 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-477zg_7e3ef95a-3fc4-4565-8c77-df73139c624f/kube-rbac-proxy/0.log" Jul 17 10:24:23.973268 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:23.973246 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-477zg_7e3ef95a-3fc4-4565-8c77-df73139c624f/exporter/0.log" Jul 17 10:24:23.997057 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:23.997038 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-477zg_7e3ef95a-3fc4-4565-8c77-df73139c624f/extractor/0.log" Jul 17 10:24:25.147727 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:25.147699 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5lvm5/perf-node-gather-daemonset-7mdsf" Jul 17 10:24:26.664865 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:26.664836 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5566fdf9-4ncqd_ce9315d2-e289-412c-b70b-3fb424690668/manager/0.log" Jul 17 10:24:26.689680 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:26.689657 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5566fdf9-4ncqd_ce9315d2-e289-412c-b70b-3fb424690668/copy-manifests/0.log" Jul 17 10:24:28.238135 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:28.238061 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7df87f8d9d-hl2cw_6672feea-572a-4101-8b3c-027c97014e5c/manager/0.log" Jul 17 10:24:28.427452 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:28.427421 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ngb4g_82b2d768-3b69-4816-9ac3-1ec68ded2216/istio-proxy/0.log" Jul 17 10:24:28.633911 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:28.633883 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m8966_a6e951b2-4383-4adf-9e5d-0291652188a3/discovery/0.log" Jul 17 10:24:30.027353 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:30.027324 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4ngb4g_82b2d768-3b69-4816-9ac3-1ec68ded2216/istio-proxy/0.log" Jul 17 10:24:30.227864 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:30.227828 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-m8966_a6e951b2-4383-4adf-9e5d-0291652188a3/discovery/0.log" Jul 17 10:24:33.307464 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:33.307422 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-zld8c_7b905e46-4bf8-45ac-9704-72dd44ba03cc/migrator/0.log" Jul 17 10:24:33.331488 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:33.331463 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-zld8c_7b905e46-4bf8-45ac-9704-72dd44ba03cc/graceful-termination/0.log" Jul 17 10:24:34.745364 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:34.745333 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4b9j2_6063385c-abf2-4e7b-91c2-5bf0d2583bb4/kube-multus/0.log" Jul 17 10:24:34.773795 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:34.773773 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mqkt_ed6be1a8-82df-4378-b517-e3e83bf03857/kube-multus-additional-cni-plugins/0.log" Jul 17 10:24:34.796097 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:34.796076 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mqkt_ed6be1a8-82df-4378-b517-e3e83bf03857/egress-router-binary-copy/0.log" Jul 17 10:24:34.819986 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:34.819970 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mqkt_ed6be1a8-82df-4378-b517-e3e83bf03857/cni-plugins/0.log" Jul 17 10:24:34.843919 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:34.843900 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mqkt_ed6be1a8-82df-4378-b517-e3e83bf03857/bond-cni-plugin/0.log" Jul 17 10:24:34.865919 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:34.865900 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mqkt_ed6be1a8-82df-4378-b517-e3e83bf03857/routeoverride-cni/0.log" Jul 17 10:24:34.886780 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:34.886758 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mqkt_ed6be1a8-82df-4378-b517-e3e83bf03857/whereabouts-cni-bincopy/0.log" Jul 17 10:24:34.909720 ip-10-0-133-236 kubenswrapper[2584]: I0717 10:24:34.909700 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8mqkt_ed6be1a8-82df-4378-b517-e3e83bf03857/whereabouts-cni/0.log"